2023-11-02T08:17:43.507 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-02T08:17:43.508 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} --name yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi --owner scheduled_yuriw@teuthology --verbose --archive /home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443892 -- /home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443892/orig.config.yaml 2023-11-02T08:17:43.533 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-02T08:17:43.609 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443892 branch: wip-yuri6-testing-2023-11-01-0745-reef description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} email: yweinste@redhat.com first_in_suite: false job_id: '7443892' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri6-testing-2023-11-01-0745-reef ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: b5a9486b4a6f144f83c51b7ec33111a2c86dc081 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: b5a9486b4a6f144f83c51b7ec33111a2c86dc081 workunit: branch: wip-yuri6-testing-2023-11-01-0745-reef sha1: b5a9486b4a6f144f83c51b7ec33111a2c86dc081 owner: scheduled_yuriw@teuthology priority: 49 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 525 sha1: b5a9486b4a6f144f83c51b7ec33111a2c86dc081 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri6-testing-2023-11-01-0745-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: b5a9486b4a6f144f83c51b7ec33111a2c86dc081 targets: smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAZePFn69drfMB7+IWqDvyKDLk1QmpU3g2Wb7klHmM2zvqrmMq/zUBEdf8SMy931EUzCcIj8XyyQm5rFSAGqn/g= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-01_21:37:41 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-11-02T08:17:43.609 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa; will attempt to use it 2023-11-02T08:17:43.610 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks 2023-11-02T08:17:43.611 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-02T08:17:43.613 INFO:teuthology.task.internal:Checking packages... 2023-11-02T08:17:43.638 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b5a9486b4a6f144f83c51b7ec33111a2c86dc081' 2023-11-02T08:17:43.639 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-02T08:17:43.639 INFO:teuthology.packaging:ref: None 2023-11-02T08:17:43.640 INFO:teuthology.packaging:tag: None 2023-11-02T08:17:43.640 INFO:teuthology.packaging:branch: wip-yuri6-testing-2023-11-01-0745-reef 2023-11-02T08:17:43.640 INFO:teuthology.packaging:sha1: b5a9486b4a6f144f83c51b7ec33111a2c86dc081 2023-11-02T08:17:43.641 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri6-testing-2023-11-01-0745-reef 2023-11-02T08:17:43.821 INFO:teuthology.task.internal:Found packages for ceph version 18.2.0-1159.gb5a9486b 2023-11-02T08:17:43.822 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-02T08:17:43.833 INFO:teuthology.task.internal:no buildpackages task found 2023-11-02T08:17:43.833 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-02T08:17:43.844 INFO:teuthology.task.internal:Saving configuration 2023-11-02T08:17:43.853 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-02T08:17:43.861 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-02T08:17:43.887 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443892', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-02 08:12:36.459141', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAZePFn69drfMB7+IWqDvyKDLk1QmpU3g2Wb7klHmM2zvqrmMq/zUBEdf8SMy931EUzCcIj8XyyQm5rFSAGqn/g='} 2023-11-02T08:17:43.887 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-02T08:17:43.897 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-11-02T08:17:43.897 INFO:teuthology.run_tasks:Running task console_log... 2023-11-02T08:17:43.944 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9a01a2ca60>, signals=[15]) 2023-11-02T08:17:43.945 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-02T08:17:43.958 INFO:teuthology.task.internal:Opening connections... 2023-11-02T08:17:43.958 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2023-11-02T08:17:43.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T08:17:44.044 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-02T08:17:44.052 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2023-11-02T08:17:44.115 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2023-11-02T08:17:44.116 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2023-11-02T08:17:44.172 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream" 2023-11-02T08:17:44.172 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="9" 2023-11-02T08:17:44.172 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2023-11-02T08:17:44.173 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2023-11-02T08:17:44.173 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="9" 2023-11-02T08:17:44.173 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el9" 2023-11-02T08:17:44.173 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-02T08:17:44.173 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2023-11-02T08:17:44.173 INFO:teuthology.orchestra.run.smithi116.stdout:LOGO="fedora-logo-icon" 2023-11-02T08:17:44.174 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-02T08:17:44.174 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2023-11-02T08:17:44.174 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-02T08:17:44.174 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-02T08:17:44.175 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-02T08:17:44.175 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2023-11-02T08:17:44.205 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-02T08:17:44.216 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-02T08:17:44.225 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-02T08:17:44.226 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2023-11-02T08:17:44.242 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-02T08:17:44.250 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-02T08:17:44.251 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2023-11-02T08:17:44.307 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-02T08:17:44.343 INFO:teuthology.run_tasks:Running task kernel... 2023-11-02T08:17:44.358 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-02T08:17:44.359 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-02T08:17:44.360 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-02T08:17:44.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T08:17:44.379 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2023-11-02T08:17:44.435 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-361.el9.x86_64 2023-11-02T08:17:44.436 INFO:teuthology.task.kernel:Running kernel on smithi116: 5.14.0-361.el9.x86_64 2023-11-02T08:17:44.436 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum install -y kernel 2023-11-02T08:17:46.606 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 11 MB/s | 6.3 MB 00:00 2023-11-02T08:17:48.209 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2023-11-02T08:17:52.097 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 7.9 MB 00:03 2023-11-02T08:17:53.939 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 22 MB/s | 18 MB 00:00 2023-11-02T08:17:58.942 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 5.8 MB 00:01 2023-11-02T08:18:00.349 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 15 kB 00:00 2023-11-02T08:18:02.000 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 13 MB/s | 19 MB 00:01 2023-11-02T08:18:06.575 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2023-11-02T08:18:08.665 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-11-02T08:18:08.666 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-11-02T08:18:08.767 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:18:08.780 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:18:08.780 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2023-11-02T08:18:08.781 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:18:08.781 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-11-02T08:18:08.782 INFO:teuthology.orchestra.run.smithi116.stdout: kernel x86_64 5.14.0-378.el9 baseos 5.3 M 2023-11-02T08:18:08.782 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2023-11-02T08:18:08.782 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core x86_64 5.14.0-378.el9 baseos 20 M 2023-11-02T08:18:08.783 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules x86_64 5.14.0-378.el9 baseos 38 M 2023-11-02T08:18:08.783 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core x86_64 5.14.0-378.el9 baseos 32 M 2023-11-02T08:18:08.783 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:18:08.784 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-02T08:18:08.785 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:18:08.785 INFO:teuthology.orchestra.run.smithi116.stdout:Install 4 Packages 2023-11-02T08:18:08.785 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:18:08.785 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 95 M 2023-11-02T08:18:08.786 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 120 M 2023-11-02T08:18:08.786 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-02T08:18:09.635 INFO:teuthology.orchestra.run.smithi116.stdout:(1/4): kernel-5.14.0-378.el9.x86_64.rpm 7.7 MB/s | 5.3 MB 00:00 2023-11-02T08:18:11.660 INFO:teuthology.orchestra.run.smithi116.stdout:(2/4): kernel-modules-5.14.0-378.el9.x86_64.rpm 14 MB/s | 38 MB 00:02 2023-11-02T08:18:13.634 INFO:teuthology.orchestra.run.smithi116.stdout:(3/4): kernel-core-5.14.0-378.el9.x86_64.rpm 4.3 MB/s | 20 MB 00:04 2023-11-02T08:18:19.168 INFO:teuthology.orchestra.run.smithi116.stdout:(4/4): kernel-modules-core-5.14.0-378.el9.x86_6 3.4 MB/s | 32 MB 00:09 2023-11-02T08:18:19.168 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:18:19.168 INFO:teuthology.orchestra.run.smithi116.stdout:Total 9.1 MB/s | 95 MB 00:10 2023-11-02T08:18:20.011 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-02T08:18:20.356 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-02T08:18:20.357 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-02T08:18:21.666 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-02T08:18:21.667 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-02T08:18:24.277 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-02T08:18:24.890 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-core-5.14.0-378.el9.x86_64 1/4 2023-11-02T08:18:25.017 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-11-02T08:18:27.761 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 2/4 2023-11-02T08:18:27.954 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-11-02T08:18:32.545 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-11-02T08:18:32.743 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-5.14.0-378.el9.x86_64 4/4 2023-11-02T08:18:37.374 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-11-02T08:19:06.063 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 4/4 2023-11-02T08:19:06.105 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 4/4 2023-11-02T08:19:06.524 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-5.14.0-378.el9.x86_64 4/4 2023-11-02T08:19:06.524 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-5.14.0-378.el9.x86_64 1/4 2023-11-02T08:19:06.525 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-11-02T08:19:06.525 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-11-02T08:19:06.919 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-11-02T08:19:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:19:06.920 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-11-02T08:19:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-5.14.0-378.el9.x86_64 2023-11-02T08:19:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core-5.14.0-378.el9.x86_64 2023-11-02T08:19:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-5.14.0-378.el9.x86_64 2023-11-02T08:19:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core-5.14.0-378.el9.x86_64 2023-11-02T08:19:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:19:06.921 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:19:07.400 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2023-11-02T08:19:07.557 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-378.el9.x86_64 2023-11-02T08:19:07.557 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-11-02T08:19:07.558 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-378.el9.x86_64 2023-11-02T08:19:07.558 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-02T08:19:07.558 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-02T08:19:07.559 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-11-02T08:19:07.559 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=None, version=distro) 2023-11-02T08:19:07.559 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2023-11-02T08:19:07.675 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-378.el9.x86_64 2023-11-02T08:19:07.676 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-11-02T08:19:07.676 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -qi grub2-tools 2023-11-02T08:19:07.713 INFO:teuthology.orchestra.run.smithi116.stdout:Name : grub2-tools 2023-11-02T08:19:07.713 INFO:teuthology.orchestra.run.smithi116.stdout:Epoch : 1 2023-11-02T08:19:07.713 INFO:teuthology.orchestra.run.smithi116.stdout:Version : 2.06 2023-11-02T08:19:07.714 INFO:teuthology.orchestra.run.smithi116.stdout:Release : 68.el9 2023-11-02T08:19:07.714 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: x86_64 2023-11-02T08:19:07.714 INFO:teuthology.orchestra.run.smithi116.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-02T08:19:07.714 INFO:teuthology.orchestra.run.smithi116.stdout:Group : Unspecified 2023-11-02T08:19:07.715 INFO:teuthology.orchestra.run.smithi116.stdout:Size : 8264254 2023-11-02T08:19:07.715 INFO:teuthology.orchestra.run.smithi116.stdout:License : GPLv3+ 2023-11-02T08:19:07.715 INFO:teuthology.orchestra.run.smithi116.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-02T08:19:07.715 INFO:teuthology.orchestra.run.smithi116.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-02T08:19:07.715 INFO:teuthology.orchestra.run.smithi116.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-02T08:19:07.716 INFO:teuthology.orchestra.run.smithi116.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-02T08:19:07.716 INFO:teuthology.orchestra.run.smithi116.stdout:Packager : builder@centos.org 2023-11-02T08:19:07.716 INFO:teuthology.orchestra.run.smithi116.stdout:Vendor : CentOS 2023-11-02T08:19:07.716 INFO:teuthology.orchestra.run.smithi116.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-02T08:19:07.716 INFO:teuthology.orchestra.run.smithi116.stdout:Summary : Support tools for GRUB. 2023-11-02T08:19:07.716 INFO:teuthology.orchestra.run.smithi116.stdout:Description : 2023-11-02T08:19:07.717 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:19:07.717 INFO:teuthology.orchestra.run.smithi116.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-02T08:19:07.717 INFO:teuthology.orchestra.run.smithi116.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-02T08:19:07.717 INFO:teuthology.orchestra.run.smithi116.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-02T08:19:07.717 INFO:teuthology.orchestra.run.smithi116.stdout:hardware devices. 2023-11-02T08:19:07.718 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:19:07.718 INFO:teuthology.orchestra.run.smithi116.stdout:This subpackage provides tools for support of all platforms. 2023-11-02T08:19:07.719 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-02T08:19:07.719 INFO:teuthology.task.kernel:Updating grub on smithi116 to boot 5.14.0-378.el9.x86_64 2023-11-02T08:19:07.720 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-02T08:19:08.311 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2023-11-02T08:19:09.675 INFO:teuthology.orchestra.run.smithi116.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-02T08:19:09.721 INFO:teuthology.orchestra.run.smithi116.stderr:done 2023-11-02T08:19:09.723 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2023-11-02T08:19:09.738 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.ao4zkdYKXK 2023-11-02T08:19:09.739 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ao4zkdYKXK 2023-11-02T08:19:09.808 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.ao4zkdYKXK 2023-11-02T08:19:09.951 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.ao4zkdYKXK is 6KB 2023-11-02T08:19:10.000 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.ao4zkdYKXK 2023-11-02T08:19:10.015 DEBUG:teuthology.orchestra.run.smithi116:> sudo /bin/ls /boot/loader/entries || true 2023-11-02T08:19:10.079 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-02T08:19:10.080 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-02T08:19:10.080 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-02T08:19:10.080 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64.conf 2023-11-02T08:19:10.081 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64 2023-11-02T08:19:10.234 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2023-11-02T08:19:40.266 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-11-02T08:19:40.267 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-11-02T08:19:40.268 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T08:19:58.714 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2023-11-02T08:20:07.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-11-02T08:20:07.723 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T08:20:10.778 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2023-11-02T08:20:22.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-11-02T08:20:22.792 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T08:20:41.210 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2023-11-02T08:20:56.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-11-02T08:20:56.227 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T08:20:56.624 DEBUG:teuthology.orchestra.run.smithi116:> true 2023-11-02T08:20:57.011 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-11-02T08:20:57.012 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-378.el9.x86_64"... 2023-11-02T08:20:57.012 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2023-11-02T08:20:57.026 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-378.el9.x86_64 2023-11-02T08:20:57.027 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-378.el9.x86_64 vs 5.14.0-378.el9.x86_64 2023-11-02T08:20:57.027 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-02T08:20:57.027 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-02T08:20:58.028 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-11-02T08:20:58.029 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-02T08:20:58.095 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1 2023-11-02T08:20:58.130 DEBUG:teuthology.parallel:result is None 2023-11-02T08:20:58.131 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-02T08:20:58.139 INFO:teuthology.task.internal:Creating test directory... 2023-11-02T08:20:58.139 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-02T08:20:58.154 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-02T08:20:58.166 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-02T08:20:58.179 INFO:teuthology.task.internal:Creating archive directory... 2023-11-02T08:20:58.180 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-02T08:20:58.223 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-02T08:20:58.232 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-02T08:20:58.233 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-02T08:20:58.281 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:20:58.293 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:20:58.295 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-02T08:20:58.303 INFO:teuthology.task.internal:Configuring sudo... 2023-11-02T08:20:58.303 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-02T08:20:58.359 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-02T08:20:58.368 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-02T08:20:58.369 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-02T08:20:58.410 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-02T08:20:58.488 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-02T08:20:58.531 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-02T08:20:58.610 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-02T08:20:58.648 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-02T08:20:58.649 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-02T08:20:58.714 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2023-11-02T08:20:58.798 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-02T08:20:58.857 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-02T08:20:58.865 INFO:teuthology.task.internal:Starting timer... 2023-11-02T08:20:58.865 INFO:teuthology.run_tasks:Running task pcp... 2023-11-02T08:20:58.897 INFO:teuthology.run_tasks:Running task selinux... 2023-11-02T08:20:58.929 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2023-11-02T08:20:58.972 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2023-11-02T08:20:58.973 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-02T08:20:58.974 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2023-11-02T08:20:58.998 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2023-11-02T08:20:58.999 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi116.front.sepia.ceph.com': 'permissive'} 2023-11-02T08:20:58.999 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-02T08:20:59.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T08:20:59.063 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-02T08:20:59.064 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2023-11-02T08:20:59.142 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-02T08:20:59.153 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-02T08:20:59.153 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-02T08:20:59.170 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-02T08:20:59.174 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi116.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-02T08:26:32.468 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi116.front.sepia.ceph.com')] 2023-11-02T08:26:32.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-11-02T08:26:32.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-02T08:26:32.556 DEBUG:teuthology.orchestra.run.smithi116:> true 2023-11-02T08:26:32.625 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-11-02T08:26:32.626 INFO:teuthology.run_tasks:Running task clock... 2023-11-02T08:26:32.638 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-02T08:26:32.639 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-02T08:26:32.640 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-02T08:26:32.697 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-02T08:26:32.715 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-02T08:26:32.743 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2023-11-02T08:26:32.757 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2023-11-02T08:26:32.773 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-02T08:26:32.790 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-02T08:26:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2023-11-02T08:26:32.836 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-02T08:26:32.836 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2023-11-02T08:26:32.837 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-02T08:26:32.837 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-02T08:26:32.837 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-02T08:26:32.837 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-02T08:26:32.838 INFO:teuthology.run_tasks:Running task install... 2023-11-02T08:26:32.845 DEBUG:teuthology.task.install:project ceph 2023-11-02T08:26:32.846 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b5a9486b4a6f144f83c51b7ec33111a2c86dc081'}} 2023-11-02T08:26:32.846 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'b5a9486b4a6f144f83c51b7ec33111a2c86dc081'} 2023-11-02T08:26:32.847 INFO:teuthology.task.install:Using flavor: default 2023-11-02T08:26:32.853 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-02T08:26:32.854 INFO:teuthology.task.install:extra packages: [] 2023-11-02T08:26:32.855 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b5a9486b4a6f144f83c51b7ec33111a2c86dc081', 'tag': None, 'wait_for_package': False} 2023-11-02T08:26:32.855 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 2023-11-02T08:26:33.038 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-11-01-0745-reef/b5a9486b4a6f144f83c51b7ec33111a2c86dc081/centos/9/flavors/default/ 2023-11-02T08:26:33.038 INFO:teuthology.task.install.rpm:Package version is 18.2.0-1159.gb5a9486b 2023-11-02T08:26:33.245 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-11-01-0745-reef/b5a9486b4a6f144f83c51b7ec33111a2c86dc081/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-11-01-0745-reef/b5a9486b4a6f144f83c51b7ec33111a2c86dc081/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-11-01-0745-reef/b5a9486b4a6f144f83c51b7ec33111a2c86dc081/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-02T08:26:33.245 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-02T08:26:33.245 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-02T08:26:33.276 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-02T08:26:33.276 DEBUG:teuthology.orchestra.run.smithi116:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b5a9486b4a6f144f83c51b7ec33111a2c86dc081/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-02T08:26:33.346 DEBUG:teuthology.orchestra.run.smithi116:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-02T08:26:33.427 DEBUG:teuthology.orchestra.run.smithi116:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-02T08:26:33.494 INFO:teuthology.orchestra.run.smithi116.stdout:check_obsoletes = 1 2023-11-02T08:26:33.496 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum clean all 2023-11-02T08:26:33.734 INFO:teuthology.orchestra.run.smithi116.stdout:72 files removed 2023-11-02T08:26:33.761 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-radosgw 2023-11-02T08:26:34.685 INFO:teuthology.orchestra.run.smithi116.stdout:ceph packages for x86_64 126 kB/s | 77 kB 00:00 2023-11-02T08:26:35.061 INFO:teuthology.orchestra.run.smithi116.stdout:ceph noarch packages 29 kB/s | 9.4 kB 00:00 2023-11-02T08:26:35.412 INFO:teuthology.orchestra.run.smithi116.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2023-11-02T08:26:35.913 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 14 MB/s | 6.3 MB 00:00 2023-11-02T08:26:37.372 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 4.7 MB/s | 2.0 MB 00:00 2023-11-02T08:26:38.254 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 7.9 MB 00:00 2023-11-02T08:26:53.871 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 18 MB 00:14 2023-11-02T08:26:59.588 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 5.8 MB 00:02 2023-11-02T08:27:00.956 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 15 kB 00:00 2023-11-02T08:27:06.340 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 3.7 MB/s | 19 MB 00:05 2023-11-02T08:27:10.765 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2023-11-02T08:27:13.118 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:27:13.131 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:27:13.131 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2023-11-02T08:27:13.131 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:27:13.132 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-11-02T08:27:13.132 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 7.7 M 2023-11-02T08:27:13.132 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2023-11-02T08:27:13.132 INFO:teuthology.orchestra.run.smithi116.stdout: librados2 x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 3.3 M 2023-11-02T08:27:13.132 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1 x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 3.0 M 2023-11-02T08:27:13.133 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2023-11-02T08:27:13.133 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options 2023-11-02T08:27:13.133 INFO:teuthology.orchestra.run.smithi116.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-02T08:27:13.133 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 5.2 M 2023-11-02T08:27:13.133 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 18 M 2023-11-02T08:27:13.134 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 24 k 2023-11-02T08:27:13.134 INFO:teuthology.orchestra.run.smithi116.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-11-02T08:27:13.134 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-02T08:27:13.134 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-02T08:27:13.135 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-02T08:27:13.135 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2 x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 659 k 2023-11-02T08:27:13.135 INFO:teuthology.orchestra.run.smithi116.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-02T08:27:13.135 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-02T08:27:13.135 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-11-02T08:27:13.136 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 475 k 2023-11-02T08:27:13.136 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-02T08:27:13.136 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2 x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 4.4 M 2023-11-02T08:27:13.136 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-02T08:27:13.136 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-02T08:27:13.137 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-02T08:27:13.137 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse 2023-11-02T08:27:13.137 INFO:teuthology.orchestra.run.smithi116.stdout: x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 45 k 2023-11-02T08:27:13.137 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 124 k 2023-11-02T08:27:13.137 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 162 k 2023-11-02T08:27:13.137 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-02T08:27:13.138 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 320 k 2023-11-02T08:27:13.138 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 298 k 2023-11-02T08:27:13.138 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 99 k 2023-11-02T08:27:13.138 INFO:teuthology.orchestra.run.smithi116.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-02T08:27:13.138 INFO:teuthology.orchestra.run.smithi116.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-02T08:27:13.139 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:27:13.139 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-02T08:27:13.139 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:27:13.139 INFO:teuthology.orchestra.run.smithi116.stdout:Install 28 Packages 2023-11-02T08:27:13.139 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 2 Packages 2023-11-02T08:27:13.140 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:27:13.140 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 53 M 2023-11-02T08:27:13.141 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-02T08:27:14.366 INFO:teuthology.orchestra.run.smithi116.stdout:(1/30): ceph-radosgw-18.2.0-1159.gb5a9486b.el9. 11 MB/s | 7.7 MB 00:00 2023-11-02T08:27:14.408 INFO:teuthology.orchestra.run.smithi116.stdout:(2/30): ceph-selinux-18.2.0-1159.gb5a9486b.el9. 581 kB/s | 24 kB 00:00 2023-11-02T08:27:14.475 INFO:teuthology.orchestra.run.smithi116.stdout:(3/30): libcephfs2-18.2.0-1159.gb5a9486b.el9.x8 9.6 MB/s | 659 kB 00:00 2023-11-02T08:27:14.525 INFO:teuthology.orchestra.run.smithi116.stdout:(4/30): libradosstriper1-18.2.0-1159.gb5a9486b. 9.3 MB/s | 475 kB 00:00 2023-11-02T08:27:14.792 INFO:teuthology.orchestra.run.smithi116.stdout:(5/30): librgw2-18.2.0-1159.gb5a9486b.el9.x86_6 17 MB/s | 4.4 MB 00:00 2023-11-02T08:27:14.842 INFO:teuthology.orchestra.run.smithi116.stdout:(6/30): python3-ceph-argparse-18.2.0-1159.gb5a9 900 kB/s | 45 kB 00:00 2023-11-02T08:27:14.893 INFO:teuthology.orchestra.run.smithi116.stdout:(7/30): python3-ceph-common-18.2.0-1159.gb5a948 2.4 MB/s | 124 kB 00:00 2023-11-02T08:27:14.943 INFO:teuthology.orchestra.run.smithi116.stdout:(8/30): python3-cephfs-18.2.0-1159.gb5a9486b.el 3.2 MB/s | 162 kB 00:00 2023-11-02T08:27:15.001 INFO:teuthology.orchestra.run.smithi116.stdout:(9/30): python3-rados-18.2.0-1159.gb5a9486b.el9 5.4 MB/s | 320 kB 00:00 2023-11-02T08:27:15.060 INFO:teuthology.orchestra.run.smithi116.stdout:(10/30): python3-rbd-18.2.0-1159.gb5a9486b.el9. 5.0 MB/s | 298 kB 00:00 2023-11-02T08:27:15.111 INFO:teuthology.orchestra.run.smithi116.stdout:(11/30): python3-rgw-18.2.0-1159.gb5a9486b.el9. 1.9 MB/s | 99 kB 00:00 2023-11-02T08:27:15.211 INFO:teuthology.orchestra.run.smithi116.stdout:(12/30): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 106 kB 00:00 2023-11-02T08:27:15.252 INFO:teuthology.orchestra.run.smithi116.stdout:(13/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2023-11-02T08:27:15.303 INFO:teuthology.orchestra.run.smithi116.stdout:(14/30): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2023-11-02T08:27:15.337 INFO:teuthology.orchestra.run.smithi116.stdout:(15/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 8.3 MB/s | 282 kB 00:00 2023-11-02T08:27:15.370 INFO:teuthology.orchestra.run.smithi116.stdout:(16/30): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2023-11-02T08:27:15.470 INFO:teuthology.orchestra.run.smithi116.stdout:(17/30): ceph-base-18.2.0-1159.gb5a9486b.el9.x8 2.9 MB/s | 5.2 MB 00:01 2023-11-02T08:27:15.563 INFO:teuthology.orchestra.run.smithi116.stdout:(18/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 236 kB/s | 45 kB 00:00 2023-11-02T08:27:15.896 INFO:teuthology.orchestra.run.smithi116.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 260 kB/s | 111 kB 00:00 2023-11-02T08:27:16.246 INFO:teuthology.orchestra.run.smithi116.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 452 kB/s | 309 kB 00:00 2023-11-02T08:27:16.297 INFO:teuthology.orchestra.run.smithi116.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 489 kB/s | 24 kB 00:00 2023-11-02T08:27:16.356 INFO:teuthology.orchestra.run.smithi116.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 833 kB/s | 49 kB 00:00 2023-11-02T08:27:16.406 INFO:teuthology.orchestra.run.smithi116.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2023-11-02T08:27:16.773 INFO:teuthology.orchestra.run.smithi116.stdout:(24/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 2.2 MB/s | 837 kB 00:00 2023-11-02T08:27:16.914 INFO:teuthology.orchestra.run.smithi116.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2023-11-02T08:27:17.390 INFO:teuthology.orchestra.run.smithi116.stdout:(26/30): thrift-0.15.0-2.el9.x86_64.rpm 3.3 MB/s | 1.6 MB 00:00 2023-11-02T08:27:17.932 INFO:teuthology.orchestra.run.smithi116.stdout:(27/30): libarrow-9.0.0-9.el9.x86_64.rpm 2.2 MB/s | 4.4 MB 00:02 2023-11-02T08:27:17.982 INFO:teuthology.orchestra.run.smithi116.stdout:(28/30): librados2-18.2.0-1159.gb5a9486b.el9.x8 5.5 MB/s | 3.3 MB 00:00 2023-11-02T08:27:18.240 INFO:teuthology.orchestra.run.smithi116.stdout:(29/30): librbd1-18.2.0-1159.gb5a9486b.el9.x86_ 9.7 MB/s | 3.0 MB 00:00 2023-11-02T08:27:19.465 INFO:teuthology.orchestra.run.smithi116.stdout:(30/30): ceph-common-18.2.0-1159.gb5a9486b.el9. 3.2 MB/s | 18 MB 00:05 2023-11-02T08:27:19.471 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:19.471 INFO:teuthology.orchestra.run.smithi116.stdout:Total 8.4 MB/s | 53 MB 00:06 2023-11-02T08:27:19.483 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-02T08:27:19.521 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-02T08:27:19.522 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-02T08:27:19.983 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-02T08:27:19.983 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-02T08:27:21.333 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-02T08:27:21.394 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-11-02T08:27:21.441 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/32 2023-11-02T08:27:21.504 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/32 2023-11-02T08:27:21.812 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-11-02T08:27:21.852 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librados2-2:18.2.0-1159.gb5a9486b.el9.x86_64 5/32 2023-11-02T08:27:21.943 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:18.2.0-1159.gb5a9486b.el9.x86_64 5/32 2023-11-02T08:27:22.025 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rados-2:18.2.0-1159.gb5a9486b.el9.x86_64 6/32 2023-11-02T08:27:22.063 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs2-2:18.2.0-1159.gb5a9486b.el9.x86_64 7/32 2023-11-02T08:27:22.131 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs2-2:18.2.0-1159.gb5a9486b.el9.x86_64 7/32 2023-11-02T08:27:22.162 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-11-02T08:27:22.215 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-argparse-2:18.2.0-1159.gb5a9486b.el9. 9/32 2023-11-02T08:27:22.280 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cephfs-2:18.2.0-1159.gb5a9486b.el9.x86_64 10/32 2023-11-02T08:27:22.306 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libradosstriper1-2:18.2.0-1159.gb5a9486b.el9.x86_6 11/32 2023-11-02T08:27:22.355 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libradosstriper1-2:18.2.0-1159.gb5a9486b.el9.x86_6 11/32 2023-11-02T08:27:22.462 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-11-02T08:27:22.498 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-11-02T08:27:22.539 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-11-02T08:27:22.570 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-11-02T08:27:22.597 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-11-02T08:27:22.633 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-11-02T08:27:22.657 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-11-02T08:27:22.956 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-11-02T08:27:22.984 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librbd1-2:18.2.0-1159.gb5a9486b.el9.x86_64 20/32 2023-11-02T08:27:23.074 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:18.2.0-1159.gb5a9486b.el9.x86_64 20/32 2023-11-02T08:27:23.105 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 21/32 2023-11-02T08:27:23.142 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-11-02T08:27:23.542 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-11-02T08:27:23.570 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librgw2-2:18.2.0-1159.gb5a9486b.el9.x86_64 24/32 2023-11-02T08:27:23.641 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librgw2-2:18.2.0-1159.gb5a9486b.el9.x86_64 24/32 2023-11-02T08:27:23.716 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 25/32 2023-11-02T08:27:23.747 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-common-2:18.2.0-1159.gb5a9486b.el9.x8 26/32 2023-11-02T08:27:25.845 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:18.2.0-1159.gb5a9486b.el9.x86_64 27/32 2023-11-02T08:27:25.877 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-common-2:18.2.0-1159.gb5a9486b.el9.x86_64 27/32 2023-11-02T08:27:26.422 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:18.2.0-1159.gb5a9486b.el9.x86_64 27/32 2023-11-02T08:27:26.497 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-base-2:18.2.0-1159.gb5a9486b.el9.x86_64 28/32 2023-11-02T08:27:26.571 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-base-2:18.2.0-1159.gb5a9486b.el9.x86_64 28/32 2023-11-02T08:27:26.571 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-02T08:27:26.572 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-02T08:27:26.572 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:27:26.608 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-selinux-2:18.2.0-1159.gb5a9486b.el9.x86_64 29/32 2023-11-02T08:27:36.042 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-selinux-2:18.2.0-1159.gb5a9486b.el9.x86_64 29/32 2023-11-02T08:27:36.042 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /sys 2023-11-02T08:27:36.043 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /proc 2023-11-02T08:27:36.043 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /mnt 2023-11-02T08:27:36.043 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /var/tmp 2023-11-02T08:27:36.043 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /home 2023-11-02T08:27:36.043 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /root 2023-11-02T08:27:36.044 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /tmp 2023-11-02T08:27:36.044 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:27:36.810 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-radosgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 30/32 2023-11-02T08:27:36.835 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-radosgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 30/32 2023-11-02T08:27:36.835 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:27:36.835 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-02T08:27:36.836 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-02T08:27:36.836 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-02T08:27:36.836 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:27:36.838 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-11-02T08:27:36.929 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-11-02T08:27:36.930 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-11-02T08:27:38.026 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-11-02T08:27:38.026 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-base-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/32 2023-11-02T08:27:38.026 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-common-2:18.2.0-1159.gb5a9486b.el9.x86_64 2/32 2023-11-02T08:27:38.026 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-radosgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 3/32 2023-11-02T08:27:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-selinux-2:18.2.0-1159.gb5a9486b.el9.x86_64 4/32 2023-11-02T08:27:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs2-2:18.2.0-1159.gb5a9486b.el9.x86_64 5/32 2023-11-02T08:27:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libradosstriper1-2:18.2.0-1159.gb5a9486b.el9.x86_6 6/32 2023-11-02T08:27:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librgw2-2:18.2.0-1159.gb5a9486b.el9.x86_64 7/32 2023-11-02T08:27:38.027 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-argparse-2:18.2.0-1159.gb5a9486b.el9. 8/32 2023-11-02T08:27:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-common-2:18.2.0-1159.gb5a9486b.el9.x8 9/32 2023-11-02T08:27:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cephfs-2:18.2.0-1159.gb5a9486b.el9.x86_64 10/32 2023-11-02T08:27:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rados-2:18.2.0-1159.gb5a9486b.el9.x86_64 11/32 2023-11-02T08:27:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 12/32 2023-11-02T08:27:38.028 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 13/32 2023-11-02T08:27:38.029 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-11-02T08:27:38.029 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-11-02T08:27:38.029 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-11-02T08:27:38.029 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-11-02T08:27:38.029 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-11-02T08:27:38.030 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-11-02T08:27:38.030 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-11-02T08:27:38.030 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-11-02T08:27:38.030 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-11-02T08:27:38.030 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-11-02T08:27:38.030 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-11-02T08:27:38.031 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-11-02T08:27:38.031 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-11-02T08:27:38.031 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-11-02T08:27:38.031 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-11-02T08:27:38.031 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:18.2.0-1159.gb5a9486b.el9.x86_64 29/32 2023-11-02T08:27:38.032 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-11-02T08:27:38.032 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:18.2.0-1159.gb5a9486b.el9.x86_64 31/32 2023-11-02T08:27:39.143 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-11-02T08:27:39.143 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:27:39.144 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2023-11-02T08:27:39.144 INFO:teuthology.orchestra.run.smithi116.stdout: librados2-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:39.144 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:39.144 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-11-02T08:27:39.144 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-02T08:27:39.145 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:39.145 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:39.145 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:39.145 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:39.145 INFO:teuthology.orchestra.run.smithi116.stdout: fmt-8.1.1-5.el9.x86_64 2023-11-02T08:27:39.146 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-02T08:27:39.146 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-02T08:27:39.146 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-02T08:27:39.146 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:39.146 INFO:teuthology.orchestra.run.smithi116.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-02T08:27:39.147 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-02T08:27:39.147 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-02T08:27:39.147 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:39.147 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-02T08:27:39.147 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:39.148 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-02T08:27:39.148 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-02T08:27:39.148 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-02T08:27:39.148 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:39.148 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:39.149 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:39.149 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-02T08:27:39.149 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:39.149 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:39.149 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:39.150 INFO:teuthology.orchestra.run.smithi116.stdout: re2-1:20211101-3.el9.x86_64 2023-11-02T08:27:39.150 INFO:teuthology.orchestra.run.smithi116.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-02T08:27:39.150 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:27:39.150 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:27:39.491 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-test 2023-11-02T08:27:40.030 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:30 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:27:40.174 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:27:40.175 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:27:40.175 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2023-11-02T08:27:40.176 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:27:40.176 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-11-02T08:27:40.176 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 40 M 2023-11-02T08:27:40.176 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2023-11-02T08:27:40.176 INFO:teuthology.orchestra.run.smithi116.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-11-02T08:27:40.177 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 166 k 2023-11-02T08:27:40.177 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-02T08:27:40.177 INFO:teuthology.orchestra.run.smithi116.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-02T08:27:40.177 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-02T08:27:40.177 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:27:40.178 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-02T08:27:40.178 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:27:40.178 INFO:teuthology.orchestra.run.smithi116.stdout:Install 6 Packages 2023-11-02T08:27:40.178 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:27:40.179 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 41 M 2023-11-02T08:27:40.179 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 167 M 2023-11-02T08:27:40.179 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-02T08:27:40.348 INFO:teuthology.orchestra.run.smithi116.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.0 MB/s | 217 kB 00:00 2023-11-02T08:27:40.440 INFO:teuthology.orchestra.run.smithi116.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.6 MB/s | 300 kB 00:00 2023-11-02T08:27:40.465 INFO:teuthology.orchestra.run.smithi116.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2023-11-02T08:27:40.616 INFO:teuthology.orchestra.run.smithi116.stdout:(4/6): libcephsqlite-18.2.0-1159.gb5a9486b.el9. 444 kB/s | 166 kB 00:00 2023-11-02T08:27:41.249 INFO:teuthology.orchestra.run.smithi116.stdout:(5/6): jq-1.6-15.el9.x86_64.rpm 238 kB/s | 187 kB 00:00 2023-11-02T08:27:44.091 INFO:teuthology.orchestra.run.smithi116.stdout:(6/6): ceph-test-18.2.0-1159.gb5a9486b.el9.x86_ 10 MB/s | 40 MB 00:03 2023-11-02T08:27:44.094 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:44.094 INFO:teuthology.orchestra.run.smithi116.stdout:Total 10 MB/s | 41 MB 00:03 2023-11-02T08:27:44.108 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-02T08:27:44.128 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-02T08:27:44.128 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-02T08:27:44.474 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-02T08:27:44.474 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-02T08:27:45.003 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-02T08:27:45.091 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-11-02T08:27:45.130 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-11-02T08:27:45.158 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-11-02T08:27:45.200 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-11-02T08:27:45.227 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephsqlite-2:18.2.0-1159.gb5a9486b.el9.x86_64 5/6 2023-11-02T08:27:48.978 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephsqlite-2:18.2.0-1159.gb5a9486b.el9.x86_64 5/6 2023-11-02T08:27:49.050 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-test-2:18.2.0-1159.gb5a9486b.el9.x86_64 6/6 2023-11-02T08:27:51.165 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-test-2:18.2.0-1159.gb5a9486b.el9.x86_64 6/6 2023-11-02T08:27:51.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-test-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/6 2023-11-02T08:27:51.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephsqlite-2:18.2.0-1159.gb5a9486b.el9.x86_64 2/6 2023-11-02T08:27:51.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-11-02T08:27:51.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2023-11-02T08:27:51.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2023-11-02T08:27:51.513 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : jq-1.6-15.el9.x86_64 6/6 2023-11-02T08:27:51.514 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:27:51.514 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-11-02T08:27:51.514 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:51.515 INFO:teuthology.orchestra.run.smithi116.stdout: jq-1.6-15.el9.x86_64 2023-11-02T08:27:51.515 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:27:51.515 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-02T08:27:51.515 INFO:teuthology.orchestra.run.smithi116.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-02T08:27:51.515 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-02T08:27:51.515 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:27:51.516 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:27:51.791 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph 2023-11-02T08:27:52.335 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:42 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:27:52.479 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:27:52.481 INFO:teuthology.orchestra.run.smithi116.stdout:========================================================================================= 2023-11-02T08:27:52.482 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2023-11-02T08:27:52.482 INFO:teuthology.orchestra.run.smithi116.stdout:========================================================================================= 2023-11-02T08:27:52.482 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-11-02T08:27:52.482 INFO:teuthology.orchestra.run.smithi116.stdout: ceph x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 6.5 k 2023-11-02T08:27:52.483 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2023-11-02T08:27:52.483 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 2.1 M 2023-11-02T08:27:52.483 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 1.4 M 2023-11-02T08:27:52.483 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 242 k 2023-11-02T08:27:52.483 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 4.4 M 2023-11-02T08:27:52.484 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 18 M 2023-11-02T08:27:52.484 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-02T08:27:52.484 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-02T08:27:52.484 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-02T08:27:52.484 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-02T08:27:52.485 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-11-02T08:27:52.485 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-02T08:27:52.485 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-02T08:27:52.485 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-02T08:27:52.485 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-02T08:27:52.486 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-02T08:27:52.486 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-02T08:27:52.486 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-02T08:27:52.486 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-02T08:27:52.486 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-02T08:27:52.487 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-02T08:27:52.487 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-02T08:27:52.487 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-02T08:27:52.487 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-02T08:27:52.487 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-02T08:27:52.488 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-11-02T08:27:52.488 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-02T08:27:52.488 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-02T08:27:52.488 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-02T08:27:52.488 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-02T08:27:52.488 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-02T08:27:52.489 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-02T08:27:52.489 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-02T08:27:52.489 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:27:52.489 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-02T08:27:52.489 INFO:teuthology.orchestra.run.smithi116.stdout:========================================================================================= 2023-11-02T08:27:52.490 INFO:teuthology.orchestra.run.smithi116.stdout:Install 33 Packages 2023-11-02T08:27:52.490 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:27:52.491 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 30 M 2023-11-02T08:27:52.491 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 109 M 2023-11-02T08:27:52.491 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-02T08:27:53.059 INFO:teuthology.orchestra.run.smithi116.stdout:(1/33): ceph-18.2.0-1159.gb5a9486b.el9.x86_64.r 39 kB/s | 6.5 kB 00:00 2023-11-02T08:27:53.434 INFO:teuthology.orchestra.run.smithi116.stdout:(2/33): ceph-mgr-18.2.0-1159.gb5a9486b.el9.x86_ 2.7 MB/s | 1.4 MB 00:00 2023-11-02T08:27:53.493 INFO:teuthology.orchestra.run.smithi116.stdout:(3/33): ceph-mon-18.2.0-1159.gb5a9486b.el9.x86_ 10 MB/s | 4.4 MB 00:00 2023-11-02T08:27:53.535 INFO:teuthology.orchestra.run.smithi116.stdout:(4/33): ceph-mds-18.2.0-1159.gb5a9486b.el9.x86_ 3.3 MB/s | 2.1 MB 00:00 2023-11-02T08:27:53.560 INFO:teuthology.orchestra.run.smithi116.stdout:(5/33): ceph-mgr-modules-core-18.2.0-1159.gb5a9 3.5 MB/s | 242 kB 00:00 2023-11-02T08:27:53.668 INFO:teuthology.orchestra.run.smithi116.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 373 kB/s | 32 kB 00:00 2023-11-02T08:27:53.694 INFO:teuthology.orchestra.run.smithi116.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 908 kB/s | 144 kB 00:00 2023-11-02T08:27:53.719 INFO:teuthology.orchestra.run.smithi116.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 771 kB/s | 39 kB 00:00 2023-11-02T08:27:53.744 INFO:teuthology.orchestra.run.smithi116.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-11-02T08:27:53.761 INFO:teuthology.orchestra.run.smithi116.stdout:(10/33): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2023-11-02T08:27:53.786 INFO:teuthology.orchestra.run.smithi116.stdout:(11/33): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2023-11-02T08:27:53.803 INFO:teuthology.orchestra.run.smithi116.stdout:(12/33): python3-requests-2.25.1-7.el9.noarch.r 2.6 MB/s | 113 kB 00:00 2023-11-02T08:27:53.820 INFO:teuthology.orchestra.run.smithi116.stdout:(13/33): python3-urllib3-1.26.5-4.el9.noarch.rp 5.4 MB/s | 187 kB 00:00 2023-11-02T08:27:54.012 INFO:teuthology.orchestra.run.smithi116.stdout:(14/33): python3-cryptography-36.0.1-4.el9.x86_ 5.9 MB/s | 1.2 MB 00:00 2023-11-02T08:27:54.321 INFO:teuthology.orchestra.run.smithi116.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 141 kB/s | 43 kB 00:00 2023-11-02T08:27:54.513 INFO:teuthology.orchestra.run.smithi116.stdout:(16/33): ceph-osd-18.2.0-1159.gb5a9486b.el9.x86 16 MB/s | 18 MB 00:01 2023-11-02T08:27:54.541 INFO:teuthology.orchestra.run.smithi116.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 792 kB/s | 172 kB 00:00 2023-11-02T08:27:54.597 INFO:teuthology.orchestra.run.smithi116.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2023-11-02T08:27:54.655 INFO:teuthology.orchestra.run.smithi116.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 302 kB/s | 18 kB 00:00 2023-11-02T08:27:54.730 INFO:teuthology.orchestra.run.smithi116.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 310 kB/s | 23 kB 00:00 2023-11-02T08:27:54.789 INFO:teuthology.orchestra.run.smithi116.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 333 kB/s | 19 kB 00:00 2023-11-02T08:27:54.843 INFO:teuthology.orchestra.run.smithi116.stdout:(22/33): python3-pytz-2021.1-5.el9.noarch.rpm 50 kB/s | 51 kB 00:01 2023-11-02T08:27:54.865 INFO:teuthology.orchestra.run.smithi116.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2023-11-02T08:27:54.898 INFO:teuthology.orchestra.run.smithi116.stdout:(24/33): python3-cherrypy-18.6.1-2.el9.noarch.r 932 kB/s | 358 kB 00:00 2023-11-02T08:27:54.949 INFO:teuthology.orchestra.run.smithi116.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 942 kB/s | 79 kB 00:00 2023-11-02T08:27:55.007 INFO:teuthology.orchestra.run.smithi116.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 282 kB/s | 16 kB 00:00 2023-11-02T08:27:55.049 INFO:teuthology.orchestra.run.smithi116.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2023-11-02T08:27:55.083 INFO:teuthology.orchestra.run.smithi116.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2023-11-02T08:27:55.108 INFO:teuthology.orchestra.run.smithi116.stdout:(29/33): python3-logutils-0.3.5-21.el9.noarch.r 172 kB/s | 46 kB 00:00 2023-11-02T08:27:55.133 INFO:teuthology.orchestra.run.smithi116.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 425 kB/s | 36 kB 00:00 2023-11-02T08:27:55.176 INFO:teuthology.orchestra.run.smithi116.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2023-11-02T08:27:55.201 INFO:teuthology.orchestra.run.smithi116.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 299 kB/s | 20 kB 00:00 2023-11-02T08:27:55.268 INFO:teuthology.orchestra.run.smithi116.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2023-11-02T08:27:55.273 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:27:55.273 INFO:teuthology.orchestra.run.smithi116.stdout:Total 11 MB/s | 30 MB 00:02 2023-11-02T08:27:55.304 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-02T08:27:55.341 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-02T08:27:55.341 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-02T08:27:55.708 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-02T08:27:55.708 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-02T08:27:56.933 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-02T08:27:56.995 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-11-02T08:27:57.035 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-11-02T08:27:57.069 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-11-02T08:27:57.159 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-11-02T08:27:57.239 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-11-02T08:27:57.279 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-11-02T08:27:57.325 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-11-02T08:27:57.392 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-11-02T08:27:57.447 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-11-02T08:27:57.485 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-11-02T08:27:57.535 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-11-02T08:27:57.598 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-11-02T08:27:57.784 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-11-02T08:27:57.840 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-11-02T08:27:57.871 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-11-02T08:27:57.905 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-11-02T08:27:57.934 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-11-02T08:27:58.010 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-11-02T08:27:58.045 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-11-02T08:27:58.095 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-11-02T08:27:58.136 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-11-02T08:27:58.171 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-11-02T08:27:58.259 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-11-02T08:27:58.302 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-11-02T08:27:58.382 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-11-02T08:27:58.519 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-11-02T08:27:58.653 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-02T08:27:58.814 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-modules-core-2:18.2.0-1159.gb5a9486b.el9. 28/33 2023-11-02T08:27:58.848 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-2:18.2.0-1159.gb5a9486b.el9.x86_64 29/33 2023-11-02T08:27:58.872 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-2:18.2.0-1159.gb5a9486b.el9.x86_64 29/33 2023-11-02T08:27:58.873 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:27:58.873 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-02T08:27:58.873 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-02T08:27:58.873 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-02T08:27:58.873 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:00.433 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-osd-2:18.2.0-1159.gb5a9486b.el9.x86_64 30/33 2023-11-02T08:28:00.461 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-osd-2:18.2.0-1159.gb5a9486b.el9.x86_64 30/33 2023-11-02T08:28:00.461 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:28:00.462 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-02T08:28:00.462 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-02T08:28:00.462 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-02T08:28:00.462 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:00.909 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mon-2:18.2.0-1159.gb5a9486b.el9.x86_64 31/33 2023-11-02T08:28:00.932 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mon-2:18.2.0-1159.gb5a9486b.el9.x86_64 31/33 2023-11-02T08:28:00.932 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:28:00.932 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-02T08:28:00.933 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-02T08:28:00.933 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-02T08:28:00.933 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:01.134 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mds-2:18.2.0-1159.gb5a9486b.el9.x86_64 32/33 2023-11-02T08:28:01.157 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mds-2:18.2.0-1159.gb5a9486b.el9.x86_64 32/33 2023-11-02T08:28:01.157 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:28:01.157 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-02T08:28:01.158 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-02T08:28:01.159 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-02T08:28:01.159 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:01.224 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-2:18.2.0-1159.gb5a9486b.el9.x86_64 33/33 2023-11-02T08:28:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-2:18.2.0-1159.gb5a9486b.el9.x86_64 33/33 2023-11-02T08:28:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/33 2023-11-02T08:28:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mds-2:18.2.0-1159.gb5a9486b.el9.x86_64 2/33 2023-11-02T08:28:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-2:18.2.0-1159.gb5a9486b.el9.x86_64 3/33 2023-11-02T08:28:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mon-2:18.2.0-1159.gb5a9486b.el9.x86_64 4/33 2023-11-02T08:28:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-osd-2:18.2.0-1159.gb5a9486b.el9.x86_64 5/33 2023-11-02T08:28:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-modules-core-2:18.2.0-1159.gb5a9486b.el9. 6/33 2023-11-02T08:28:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-11-02T08:28:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-11-02T08:28:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-11-02T08:28:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-11-02T08:28:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-11-02T08:28:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-11-02T08:28:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-11-02T08:28:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-11-02T08:28:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-11-02T08:28:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-11-02T08:28:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-11-02T08:28:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-11-02T08:28:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-11-02T08:28:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-11-02T08:28:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-11-02T08:28:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-11-02T08:28:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-11-02T08:28:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-11-02T08:28:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-11-02T08:28:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-11-02T08:28:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-02T08:28:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-11-02T08:28:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-11-02T08:28:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-11-02T08:28:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-11-02T08:28:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-11-02T08:28:03.958 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-11-02T08:28:03.958 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:03.959 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-11-02T08:28:03.959 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:28:03.959 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:28:03.960 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:28:03.960 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:28:03.960 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:28:03.961 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:28:03.961 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-02T08:28:03.961 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-02T08:28:03.962 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-02T08:28:03.962 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-02T08:28:03.963 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-02T08:28:03.963 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-02T08:28:03.964 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-02T08:28:03.964 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-02T08:28:03.964 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-02T08:28:03.965 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-02T08:28:03.965 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-02T08:28:03.965 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-02T08:28:03.966 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-02T08:28:03.967 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-02T08:28:03.967 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-02T08:28:03.968 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply-3.11-14.el9.noarch 2023-11-02T08:28:03.968 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-02T08:28:03.969 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-02T08:28:03.969 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-02T08:28:03.969 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-02T08:28:03.970 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-02T08:28:03.970 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-02T08:28:03.970 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-02T08:28:03.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-02T08:28:03.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-02T08:28:03.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-02T08:28:03.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-02T08:28:03.972 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:03.973 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:04.304 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-base 2023-11-02T08:28:04.845 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:54 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:04.936 INFO:teuthology.orchestra.run.smithi116.stdout:Package ceph-base-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:28:04.984 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:04.985 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2023-11-02T08:28:04.985 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:05.052 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install cephadm 2023-11-02T08:28:05.585 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:55 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:05.723 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:05.724 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2023-11-02T08:28:05.725 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:05.725 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-11-02T08:28:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 221 k 2023-11-02T08:28:05.725 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:05.726 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-02T08:28:05.726 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:05.726 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2023-11-02T08:28:05.726 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:05.726 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 221 k 2023-11-02T08:28:05.727 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 213 k 2023-11-02T08:28:05.727 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-02T08:28:06.457 INFO:teuthology.orchestra.run.smithi116.stdout:cephadm-18.2.0-1159.gb5a9486b.el9.noarch.rpm 303 kB/s | 221 kB 00:00 2023-11-02T08:28:06.458 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:28:06.458 INFO:teuthology.orchestra.run.smithi116.stdout:Total 302 kB/s | 221 kB 00:00 2023-11-02T08:28:06.458 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-02T08:28:06.463 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-02T08:28:06.464 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-02T08:28:06.467 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-02T08:28:06.467 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-02T08:28:06.537 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-02T08:28:06.967 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:28:07.053 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:28:07.436 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:28:07.731 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:28:07.731 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:07.731 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-11-02T08:28:07.732 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:28:07.732 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:07.732 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:07.943 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-immutable-object-cache 2023-11-02T08:28:08.494 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:58 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:08.638 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:08.639 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:08.640 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2023-11-02T08:28:08.640 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:08.640 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-11-02T08:28:08.640 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 142 k 2023-11-02T08:28:08.640 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:08.641 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-02T08:28:08.641 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:08.641 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2023-11-02T08:28:08.641 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:08.641 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 142 k 2023-11-02T08:28:08.642 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 400 k 2023-11-02T08:28:08.642 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-02T08:28:09.246 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-immutable-object-cache-18.2.0-1159.gb5a948 235 kB/s | 142 kB 00:00 2023-11-02T08:28:09.250 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:28:09.250 INFO:teuthology.orchestra.run.smithi116.stdout:Total 235 kB/s | 142 kB 00:00 2023-11-02T08:28:09.250 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-02T08:28:09.255 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-02T08:28:09.255 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-02T08:28:09.306 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-02T08:28:09.307 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-02T08:28:09.462 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-02T08:28:09.514 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-immutable-object-cache-2:18.2.0-1159.gb5a9486b. 1/1 2023-11-02T08:28:09.538 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.0-1159.gb5a9486b. 1/1 2023-11-02T08:28:09.538 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:28:09.538 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-02T08:28:09.539 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:10.526 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-immutable-object-cache-2:18.2.0-1159.gb5a9486b. 1/1 2023-11-02T08:28:10.559 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:10.559 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-11-02T08:28:10.560 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:28:10.560 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:10.560 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:10.748 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr 2023-11-02T08:28:11.299 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:01 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:11.390 INFO:teuthology.orchestra.run.smithi116.stdout:Package ceph-mgr-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:28:11.438 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:11.439 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2023-11-02T08:28:11.439 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:11.511 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-dashboard 2023-11-02T08:28:12.049 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:02 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:12.189 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:12.189 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:12.190 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2023-11-02T08:28:12.190 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:12.190 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-11-02T08:28:12.190 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 1.8 M 2023-11-02T08:28:12.190 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2023-11-02T08:28:12.191 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 23 k 2023-11-02T08:28:12.191 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 15 k 2023-11-02T08:28:12.191 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-02T08:28:12.191 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-02T08:28:12.191 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-02T08:28:12.192 INFO:teuthology.orchestra.run.smithi116.stdout:Installing weak dependencies: 2023-11-02T08:28:12.192 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-02T08:28:12.192 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:12.192 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-02T08:28:12.192 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:12.193 INFO:teuthology.orchestra.run.smithi116.stdout:Install 7 Packages 2023-11-02T08:28:12.193 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:12.193 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 2.1 M 2023-11-02T08:28:12.194 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 8.5 M 2023-11-02T08:28:12.194 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-02T08:28:12.708 INFO:teuthology.orchestra.run.smithi116.stdout:(1/7): ceph-prometheus-alerts-18.2.0-1159.gb5a9 33 kB/s | 15 kB 00:00 2023-11-02T08:28:12.758 INFO:teuthology.orchestra.run.smithi116.stdout:(2/7): ceph-grafana-dashboards-18.2.0-1159.gb5a 47 kB/s | 23 kB 00:00 2023-11-02T08:28:12.917 INFO:teuthology.orchestra.run.smithi116.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 43 kB/s | 9.0 kB 00:00 2023-11-02T08:28:13.126 INFO:teuthology.orchestra.run.smithi116.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 147 kB/s | 31 kB 00:00 2023-11-02T08:28:13.184 INFO:teuthology.orchestra.run.smithi116.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 96 kB/s | 41 kB 00:00 2023-11-02T08:28:13.310 INFO:teuthology.orchestra.run.smithi116.stdout:(6/7): ceph-mgr-dashboard-18.2.0-1159.gb5a9486b 1.7 MB/s | 1.8 MB 00:01 2023-11-02T08:28:13.426 INFO:teuthology.orchestra.run.smithi116.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 628 kB/s | 188 kB 00:00 2023-11-02T08:28:13.428 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:28:13.429 INFO:teuthology.orchestra.run.smithi116.stdout:Total 1.7 MB/s | 2.1 MB 00:01 2023-11-02T08:28:13.429 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-02T08:28:13.438 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-02T08:28:13.438 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-02T08:28:13.504 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-02T08:28:13.505 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-02T08:28:13.809 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-02T08:28:13.878 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-11-02T08:28:13.912 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-11-02T08:28:13.958 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-11-02T08:28:13.998 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-11-02T08:28:14.027 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-prometheus-alerts-2:18.2.0-1159.gb5a9486b.el9.n 5/7 2023-11-02T08:28:14.261 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-grafana-dashboards-2:18.2.0-1159.gb5a9486b.el9. 6/7 2023-11-02T08:28:14.297 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-dashboard-2:18.2.0-1159.gb5a9486b.el9.noarc 7/7 2023-11-02T08:28:14.794 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.0-1159.gb5a9486b.el9.noarc 7/7 2023-11-02T08:28:14.794 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-grafana-dashboards-2:18.2.0-1159.gb5a9486b.el9. 1/7 2023-11-02T08:28:14.794 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-dashboard-2:18.2.0-1159.gb5a9486b.el9.noarc 2/7 2023-11-02T08:28:14.795 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-prometheus-alerts-2:18.2.0-1159.gb5a9486b.el9.n 3/7 2023-11-02T08:28:14.795 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-11-02T08:28:14.795 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-11-02T08:28:14.795 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-11-02T08:28:15.091 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-11-02T08:28:15.091 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:15.091 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-11-02T08:28:15.092 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:28:15.092 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:28:15.092 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:28:15.092 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-02T08:28:15.093 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-02T08:28:15.093 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-02T08:28:15.093 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-02T08:28:15.093 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:15.093 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:15.298 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-02T08:28:15.833 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:05 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:15.973 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:15.974 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2023-11-02T08:28:15.975 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2023-11-02T08:28:15.975 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2023-11-02T08:28:15.975 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-11-02T08:28:15.975 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 7.4 M 2023-11-02T08:28:15.976 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2023-11-02T08:28:15.976 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-02T08:28:15.976 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-02T08:28:15.976 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-02T08:28:15.976 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-02T08:28:15.977 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-02T08:28:15.977 INFO:teuthology.orchestra.run.smithi116.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-02T08:28:15.977 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-02T08:28:15.977 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-11-02T08:28:15.977 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-02T08:28:15.978 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-02T08:28:15.978 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-02T08:28:15.978 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:15.978 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-02T08:28:15.979 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2023-11-02T08:28:15.979 INFO:teuthology.orchestra.run.smithi116.stdout:Install 12 Packages 2023-11-02T08:28:15.979 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:15.980 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 37 M 2023-11-02T08:28:15.980 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 209 M 2023-11-02T08:28:15.980 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-02T08:28:16.116 INFO:teuthology.orchestra.run.smithi116.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 227 kB/s | 32 kB 00:00 2023-11-02T08:28:16.149 INFO:teuthology.orchestra.run.smithi116.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 518 kB/s | 17 kB 00:00 2023-11-02T08:28:16.183 INFO:teuthology.orchestra.run.smithi116.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.1 MB/s | 34 kB 00:00 2023-11-02T08:28:16.233 INFO:teuthology.orchestra.run.smithi116.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-11-02T08:28:16.301 INFO:teuthology.orchestra.run.smithi116.stdout:(5/12): python3-devel-3.9.17-2.el9.x86_64.rpm 3.0 MB/s | 205 kB 00:00 2023-11-02T08:28:16.369 INFO:teuthology.orchestra.run.smithi116.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2023-11-02T08:28:16.443 INFO:teuthology.orchestra.run.smithi116.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2023-11-02T08:28:16.468 INFO:teuthology.orchestra.run.smithi116.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-11-02T08:28:16.535 INFO:teuthology.orchestra.run.smithi116.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 12 MB/s | 803 kB 00:00 2023-11-02T08:28:16.719 INFO:teuthology.orchestra.run.smithi116.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 57 MB/s | 16 MB 00:00 2023-11-02T08:28:16.744 INFO:teuthology.orchestra.run.smithi116.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 946 kB/s | 195 kB 00:00 2023-11-02T08:28:18.969 INFO:teuthology.orchestra.run.smithi116.stdout:(12/12): ceph-mgr-diskprediction-local-18.2.0-1 2.5 MB/s | 7.4 MB 00:02 2023-11-02T08:28:18.972 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:28:18.973 INFO:teuthology.orchestra.run.smithi116.stdout:Total 12 MB/s | 37 MB 00:02 2023-11-02T08:28:18.973 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-02T08:28:18.998 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-02T08:28:18.998 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-02T08:28:19.431 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-02T08:28:19.432 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-02T08:28:20.183 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-02T08:28:20.272 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-11-02T08:28:20.298 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-11-02T08:28:20.391 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-11-02T08:28:20.434 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-11-02T08:28:20.605 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-02T08:28:20.634 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-02T08:28:20.713 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-11-02T08:28:21.365 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-11-02T08:28:21.479 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-11-02T08:28:22.702 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-11-02T08:28:23.737 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-11-02T08:28:23.766 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.0-1159.gb5a94 12/12 2023-11-02T08:28:26.177 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.0-1159.gb5a94 12/12 2023-11-02T08:28:26.177 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.0-1159.gb5a94 1/12 2023-11-02T08:28:26.178 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-11-02T08:28:26.178 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-11-02T08:28:26.178 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-11-02T08:28:26.178 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-02T08:28:26.179 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-02T08:28:26.179 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-11-02T08:28:26.179 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-11-02T08:28:26.179 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-11-02T08:28:26.180 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-11-02T08:28:26.180 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-11-02T08:28:26.637 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-11-02T08:28:26.637 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:26.637 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-11-02T08:28:26.638 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:28:26.638 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-02T08:28:26.638 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-02T08:28:26.638 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-02T08:28:26.639 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-02T08:28:26.639 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-02T08:28:26.639 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-02T08:28:26.639 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-02T08:28:26.639 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-02T08:28:26.640 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-02T08:28:26.640 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-02T08:28:26.640 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-02T08:28:26.640 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:26.640 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:27.010 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-rook 2023-11-02T08:28:27.562 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:17 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:27.704 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:27.705 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================== 2023-11-02T08:28:27.705 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2023-11-02T08:28:27.706 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================== 2023-11-02T08:28:27.706 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-11-02T08:28:27.706 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 47 k 2023-11-02T08:28:27.706 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2023-11-02T08:28:27.706 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-02T08:28:27.707 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-02T08:28:27.707 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-11-02T08:28:27.707 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-02T08:28:27.707 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-02T08:28:27.708 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-02T08:28:27.708 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-02T08:28:27.708 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-02T08:28:27.708 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-02T08:28:27.708 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-02T08:28:27.708 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-02T08:28:27.709 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-02T08:28:27.709 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:27.709 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-02T08:28:27.709 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================== 2023-11-02T08:28:27.709 INFO:teuthology.orchestra.run.smithi116.stdout:Install 13 Packages 2023-11-02T08:28:27.710 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:27.710 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 2.0 M 2023-11-02T08:28:27.710 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 26 M 2023-11-02T08:28:27.711 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-02T08:28:28.257 INFO:teuthology.orchestra.run.smithi116.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 202 kB/s | 24 kB 00:00 2023-11-02T08:28:28.282 INFO:teuthology.orchestra.run.smithi116.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 124 kB/s | 18 kB 00:00 2023-11-02T08:28:28.324 INFO:teuthology.orchestra.run.smithi116.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-11-02T08:28:28.349 INFO:teuthology.orchestra.run.smithi116.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2023-11-02T08:28:28.375 INFO:teuthology.orchestra.run.smithi116.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-11-02T08:28:28.400 INFO:teuthology.orchestra.run.smithi116.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 850 kB/s | 43 kB 00:00 2023-11-02T08:28:28.484 INFO:teuthology.orchestra.run.smithi116.stdout:(7/13): ceph-mgr-rook-18.2.0-1159.gb5a9486b.el9 137 kB/s | 47 kB 00:00 2023-11-02T08:28:28.509 INFO:teuthology.orchestra.run.smithi116.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 138 kB/s | 15 kB 00:00 2023-11-02T08:28:28.534 INFO:teuthology.orchestra.run.smithi116.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 202 kB/s | 32 kB 00:00 2023-11-02T08:28:28.735 INFO:teuthology.orchestra.run.smithi116.stdout:(10/13): python3-google-auth-2.23.3-1.el9.noarc 786 kB/s | 197 kB 00:00 2023-11-02T08:28:28.760 INFO:teuthology.orchestra.run.smithi116.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 262 kB/s | 59 kB 00:00 2023-11-02T08:28:28.852 INFO:teuthology.orchestra.run.smithi116.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 768 kB/s | 90 kB 00:00 2023-11-02T08:28:29.210 INFO:teuthology.orchestra.run.smithi116.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2023-11-02T08:28:29.212 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:28:29.213 INFO:teuthology.orchestra.run.smithi116.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2023-11-02T08:28:29.213 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-02T08:28:29.227 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-02T08:28:29.228 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-02T08:28:29.330 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-02T08:28:29.331 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-02T08:28:29.841 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-02T08:28:30.007 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-11-02T08:28:30.060 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-11-02T08:28:30.116 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-11-02T08:28:30.145 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-11-02T08:28:30.186 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-11-02T08:28:30.286 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-11-02T08:28:30.387 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-11-02T08:28:30.438 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-11-02T08:28:31.166 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-11-02T08:28:31.210 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-11-02T08:28:31.244 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-11-02T08:28:31.289 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-11-02T08:28:31.314 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-rook-2:18.2.0-1159.gb5a9486b.el9.noarch 13/13 2023-11-02T08:28:32.116 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.0-1159.gb5a9486b.el9.noarch 13/13 2023-11-02T08:28:32.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-rook-2:18.2.0-1159.gb5a9486b.el9.noarch 1/13 2023-11-02T08:28:32.116 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-11-02T08:28:32.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-11-02T08:28:32.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-11-02T08:28:32.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-11-02T08:28:32.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-11-02T08:28:32.117 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-11-02T08:28:32.118 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-11-02T08:28:32.118 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-11-02T08:28:32.118 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-11-02T08:28:32.118 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-11-02T08:28:32.118 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-11-02T08:28:32.490 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-11-02T08:28:32.490 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:32.491 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-11-02T08:28:32.491 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:28:32.491 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-02T08:28:32.491 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-02T08:28:32.491 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-11-02T08:28:32.492 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-02T08:28:32.492 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-02T08:28:32.492 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-02T08:28:32.492 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-02T08:28:32.492 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-02T08:28:32.493 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-02T08:28:32.493 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-02T08:28:32.493 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-02T08:28:32.493 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-02T08:28:32.494 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:32.494 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:32.742 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-cephadm 2023-11-02T08:28:33.293 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:23 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:33.438 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:33.438 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================== 2023-11-02T08:28:33.438 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2023-11-02T08:28:33.438 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================== 2023-11-02T08:28:33.439 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-11-02T08:28:33.439 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 132 k 2023-11-02T08:28:33.439 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2023-11-02T08:28:33.439 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-02T08:28:33.439 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-02T08:28:33.440 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-02T08:28:33.440 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-02T08:28:33.440 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-02T08:28:33.440 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:33.440 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-02T08:28:33.441 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================== 2023-11-02T08:28:33.441 INFO:teuthology.orchestra.run.smithi116.stdout:Install 6 Packages 2023-11-02T08:28:33.441 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:33.441 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 6.8 M 2023-11-02T08:28:33.441 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 33 M 2023-11-02T08:28:33.441 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-02T08:28:33.638 INFO:teuthology.orchestra.run.smithi116.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.5 MB/s | 228 kB 00:00 2023-11-02T08:28:33.722 INFO:teuthology.orchestra.run.smithi116.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 25 MB/s | 5.8 MB 00:00 2023-11-02T08:28:33.822 INFO:teuthology.orchestra.run.smithi116.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 584 kB/s | 58 kB 00:00 2023-11-02T08:28:33.872 INFO:teuthology.orchestra.run.smithi116.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2023-11-02T08:28:33.897 INFO:teuthology.orchestra.run.smithi116.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 2.1 MB/s | 548 kB 00:00 2023-11-02T08:28:34.048 INFO:teuthology.orchestra.run.smithi116.stdout:(6/6): ceph-mgr-cephadm-18.2.0-1159.gb5a9486b.e 236 kB/s | 132 kB 00:00 2023-11-02T08:28:34.049 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:28:34.049 INFO:teuthology.orchestra.run.smithi116.stdout:Total 11 MB/s | 6.8 MB 00:00 2023-11-02T08:28:34.050 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-02T08:28:34.059 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-02T08:28:34.059 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-02T08:28:34.147 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-02T08:28:34.147 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-02T08:28:34.441 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-02T08:28:34.605 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-11-02T08:28:34.658 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-11-02T08:28:35.168 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-11-02T08:28:35.252 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-11-02T08:28:35.313 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-11-02T08:28:35.336 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 6/6 2023-11-02T08:28:36.124 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 6/6 2023-11-02T08:28:36.124 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 1/6 2023-11-02T08:28:36.125 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-11-02T08:28:36.125 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-11-02T08:28:36.125 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-11-02T08:28:36.125 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-11-02T08:28:36.423 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-11-02T08:28:36.423 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:36.423 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-11-02T08:28:36.423 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:28:36.424 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-02T08:28:36.424 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-02T08:28:36.424 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-02T08:28:36.424 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-02T08:28:36.424 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-02T08:28:36.425 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:36.425 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:36.688 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-fuse 2023-11-02T08:28:37.230 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:27 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:37.367 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:37.368 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:37.369 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2023-11-02T08:28:37.369 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:37.369 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-11-02T08:28:37.369 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 840 k 2023-11-02T08:28:37.369 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2023-11-02T08:28:37.370 INFO:teuthology.orchestra.run.smithi116.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-02T08:28:37.370 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:37.370 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-02T08:28:37.370 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:37.370 INFO:teuthology.orchestra.run.smithi116.stdout:Install 2 Packages 2023-11-02T08:28:37.371 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:37.371 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 919 k 2023-11-02T08:28:37.371 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 2.7 M 2023-11-02T08:28:37.371 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-02T08:28:37.483 INFO:teuthology.orchestra.run.smithi116.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 696 kB/s | 79 kB 00:00 2023-11-02T08:28:37.691 INFO:teuthology.orchestra.run.smithi116.stdout:(2/2): ceph-fuse-18.2.0-1159.gb5a9486b.el9.x86_ 2.6 MB/s | 840 kB 00:00 2023-11-02T08:28:37.692 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:28:37.693 INFO:teuthology.orchestra.run.smithi116.stdout:Total 2.8 MB/s | 919 kB 00:00 2023-11-02T08:28:37.693 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-02T08:28:37.713 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-02T08:28:37.713 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-02T08:28:37.788 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-02T08:28:37.788 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-02T08:28:37.972 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-02T08:28:38.097 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-11-02T08:28:38.165 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 2/2 2023-11-02T08:28:38.830 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 2/2 2023-11-02T08:28:38.830 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/2 2023-11-02T08:28:39.127 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-11-02T08:28:39.128 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:39.128 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-11-02T08:28:39.128 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-11-02T08:28:39.128 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:39.128 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:39.360 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-volume 2023-11-02T08:28:39.907 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:29 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:40.048 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:40.049 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:40.049 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2023-11-02T08:28:40.050 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:40.050 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-11-02T08:28:40.050 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume noarch 2:18.2.0-1159.gb5a9486b.el9 ceph-noarch 262 k 2023-11-02T08:28:40.050 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:40.051 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-02T08:28:40.051 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:40.051 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2023-11-02T08:28:40.051 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:40.051 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 262 k 2023-11-02T08:28:40.052 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 1.2 M 2023-11-02T08:28:40.052 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-02T08:28:40.278 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-volume-18.2.0-1159.gb5a9486b.el9.noarch.rp 1.1 MB/s | 262 kB 00:00 2023-11-02T08:28:40.279 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:28:40.280 INFO:teuthology.orchestra.run.smithi116.stdout:Total 1.1 MB/s | 262 kB 00:00 2023-11-02T08:28:40.280 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-02T08:28:40.287 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-02T08:28:40.288 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-02T08:28:40.322 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-02T08:28:40.323 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-02T08:28:40.535 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-02T08:28:40.605 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-volume-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:28:40.621 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-volume-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:28:40.622 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:28:40.622 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-02T08:28:40.622 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:41.696 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-volume-2:18.2.0-1159.gb5a9486b.el9.noarch 1/1 2023-11-02T08:28:41.696 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:41.697 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-11-02T08:28:41.697 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume-2:18.2.0-1159.gb5a9486b.el9.noarch 2023-11-02T08:28:41.697 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:41.697 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:42.008 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librados-devel 2023-11-02T08:28:42.571 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:32 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:42.713 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:42.714 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:42.714 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2023-11-02T08:28:42.715 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:42.715 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-11-02T08:28:42.715 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 127 k 2023-11-02T08:28:42.715 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:42.715 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-02T08:28:42.716 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:42.716 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2023-11-02T08:28:42.716 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:42.716 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 127 k 2023-11-02T08:28:42.716 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 456 k 2023-11-02T08:28:42.717 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-02T08:28:42.941 INFO:teuthology.orchestra.run.smithi116.stdout:librados-devel-18.2.0-1159.gb5a9486b.el9.x86_64 560 kB/s | 127 kB 00:00 2023-11-02T08:28:42.942 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:28:42.943 INFO:teuthology.orchestra.run.smithi116.stdout:Total 556 kB/s | 127 kB 00:00 2023-11-02T08:28:42.943 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-02T08:28:42.948 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-02T08:28:42.948 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-02T08:28:43.003 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-02T08:28:43.003 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-02T08:28:43.150 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-02T08:28:43.239 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librados-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:28:43.675 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:28:43.986 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:28:43.987 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:43.987 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-11-02T08:28:43.987 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:28:43.988 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:43.988 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:44.231 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install libcephfs2 2023-11-02T08:28:44.788 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:34 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:44.883 INFO:teuthology.orchestra.run.smithi116.stdout:Package libcephfs2-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:28:44.933 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:44.934 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2023-11-02T08:28:44.935 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:44.998 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install libcephfs-devel 2023-11-02T08:28:45.541 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:35 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:45.681 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:45.682 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:45.682 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2023-11-02T08:28:45.682 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:45.683 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-11-02T08:28:45.683 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 31 k 2023-11-02T08:28:45.683 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:45.683 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-02T08:28:45.683 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:45.684 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2023-11-02T08:28:45.684 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:45.684 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 31 k 2023-11-02T08:28:45.684 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 139 k 2023-11-02T08:28:45.684 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-02T08:28:45.862 INFO:teuthology.orchestra.run.smithi116.stdout:libcephfs-devel-18.2.0-1159.gb5a9486b.el9.x86_6 173 kB/s | 31 kB 00:00 2023-11-02T08:28:45.862 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:28:45.863 INFO:teuthology.orchestra.run.smithi116.stdout:Total 172 kB/s | 31 kB 00:00 2023-11-02T08:28:45.863 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-02T08:28:45.866 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-02T08:28:45.866 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-02T08:28:45.894 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-02T08:28:45.894 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-02T08:28:45.978 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-02T08:28:46.093 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:28:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:28:46.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:28:46.736 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:46.736 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-11-02T08:28:46.736 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:28:46.736 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:46.737 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:46.920 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librados2 2023-11-02T08:28:47.467 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:37 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:47.559 INFO:teuthology.orchestra.run.smithi116.stdout:Package librados2-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:28:47.609 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:47.609 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2023-11-02T08:28:47.610 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:47.671 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librbd1 2023-11-02T08:28:48.215 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:38 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:48.308 INFO:teuthology.orchestra.run.smithi116.stdout:Package librbd1-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:28:48.357 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:48.358 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2023-11-02T08:28:48.359 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:48.439 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rados 2023-11-02T08:28:48.980 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:38 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:49.073 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rados-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:28:49.123 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:49.124 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2023-11-02T08:28:49.124 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:49.185 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rgw 2023-11-02T08:28:49.728 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:39 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:49.819 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rgw-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:28:49.868 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:49.869 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2023-11-02T08:28:49.869 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:49.928 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-cephfs 2023-11-02T08:28:50.474 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:40 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:50.565 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-cephfs-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:28:50.614 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:50.615 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2023-11-02T08:28:50.615 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:50.684 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rbd 2023-11-02T08:28:51.229 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:41 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:51.320 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 is already installed. 2023-11-02T08:28:51.370 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:51.371 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2023-11-02T08:28:51.371 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:51.436 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-fuse 2023-11-02T08:28:51.977 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:41 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:52.115 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:52.116 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:52.116 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2023-11-02T08:28:52.117 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:52.117 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-11-02T08:28:52.117 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 86 k 2023-11-02T08:28:52.118 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:52.118 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-02T08:28:52.118 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:52.119 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2023-11-02T08:28:52.119 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:52.119 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 86 k 2023-11-02T08:28:52.120 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 231 k 2023-11-02T08:28:52.120 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-02T08:28:52.431 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-fuse-18.2.0-1159.gb5a9486b.el9.x86_64.rpm 274 kB/s | 86 kB 00:00 2023-11-02T08:28:52.431 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:28:52.432 INFO:teuthology.orchestra.run.smithi116.stdout:Total 273 kB/s | 86 kB 00:00 2023-11-02T08:28:52.432 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-02T08:28:52.438 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-02T08:28:52.439 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-02T08:28:52.493 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-02T08:28:52.494 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-02T08:28:52.628 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-02T08:28:52.720 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:28:53.095 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:28:53.401 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:28:53.401 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:53.402 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-11-02T08:28:53.402 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:28:53.402 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:53.402 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:53.623 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-mirror 2023-11-02T08:28:54.174 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:44 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:54.320 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:54.321 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:54.321 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2023-11-02T08:28:54.321 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:54.322 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-11-02T08:28:54.322 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 3.0 M 2023-11-02T08:28:54.322 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:54.322 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-02T08:28:54.323 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:54.323 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2023-11-02T08:28:54.323 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:54.323 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 3.0 M 2023-11-02T08:28:54.323 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 12 M 2023-11-02T08:28:54.324 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-02T08:28:54.683 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-mirror-18.2.0-1159.gb5a9486b.el9.x86_64.rpm 8.2 MB/s | 3.0 MB 00:00 2023-11-02T08:28:54.684 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:28:54.685 INFO:teuthology.orchestra.run.smithi116.stdout:Total 8.2 MB/s | 3.0 MB 00:00 2023-11-02T08:28:54.685 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-02T08:28:54.694 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-02T08:28:54.694 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-02T08:28:54.765 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-02T08:28:54.765 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-02T08:28:55.165 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-02T08:28:55.226 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-mirror-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:28:55.250 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-mirror-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:28:55.250 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-02T08:28:55.251 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-02T08:28:55.251 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-02T08:28:55.251 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-02T08:28:55.251 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:56.393 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-mirror-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:28:56.393 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:56.393 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-11-02T08:28:56.394 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:28:56.394 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:56.394 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:56.621 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-nbd 2023-11-02T08:28:57.186 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:47 ago on Thu 02 Nov 2023 08:27:10 AM UTC. 2023-11-02T08:28:57.326 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-02T08:28:57.327 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:57.327 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2023-11-02T08:28:57.327 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:57.327 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-11-02T08:28:57.328 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd x86_64 2:18.2.0-1159.gb5a9486b.el9 ceph 170 k 2023-11-02T08:28:57.328 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:57.328 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-02T08:28:57.328 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-02T08:28:57.328 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2023-11-02T08:28:57.329 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:57.329 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 170 k 2023-11-02T08:28:57.329 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 490 k 2023-11-02T08:28:57.329 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-02T08:28:57.553 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-nbd-18.2.0-1159.gb5a9486b.el9.x86_64.rpm 753 kB/s | 170 kB 00:00 2023-11-02T08:28:57.554 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-02T08:28:57.554 INFO:teuthology.orchestra.run.smithi116.stdout:Total 748 kB/s | 170 kB 00:00 2023-11-02T08:28:57.554 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-02T08:28:57.562 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-02T08:28:57.562 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-02T08:28:57.621 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-02T08:28:57.621 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-02T08:28:57.763 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-02T08:28:57.860 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-nbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:28:58.285 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-nbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:28:58.626 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-nbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 1/1 2023-11-02T08:28:58.626 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:58.627 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-11-02T08:28:58.627 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd-2:18.2.0-1159.gb5a9486b.el9.x86_64 2023-11-02T08:28:58.627 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-02T08:28:58.627 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-02T08:28:58.862 DEBUG:teuthology.parallel:result is None 2023-11-02T08:28:58.863 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 2023-11-02T08:28:59.042 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-02T08:28:59.062 INFO:teuthology.orchestra.run.smithi116.stdout:18.2.0-1159.gb5a9486b.el9 2023-11-02T08:28:59.063 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-1159.gb5a9486b.el9 2023-11-02T08:28:59.063 INFO:teuthology.task.install:The correct ceph version 18.2.0-1159.gb5a9486b is installed. 2023-11-02T08:28:59.064 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-02T08:28:59.064 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-02T08:28:59.065 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-02T08:28:59.130 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-02T08:28:59.131 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-02T08:28:59.131 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2023-11-02T08:28:59.196 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-02T08:28:59.262 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-02T08:28:59.263 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-02T08:28:59.263 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-02T08:28:59.329 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-02T08:28:59.395 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-02T08:28:59.396 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-02T08:28:59.396 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2023-11-02T08:28:59.462 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-02T08:28:59.528 INFO:teuthology.run_tasks:Running task workunit... 2023-11-02T08:28:59.542 INFO:tasks.workunit:Pulling workunits from ref b5a9486b4a6f144f83c51b7ec33111a2c86dc081 2023-11-02T08:28:59.543 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-11-02T08:28:59.543 INFO:tasks.workunit:timeout=3h 2023-11-02T08:28:59.544 INFO:tasks.workunit:cleanup=True 2023-11-02T08:28:59.545 DEBUG:teuthology.orchestra.run.smithi116:> stat -- /home/ubuntu/cephtest/mnt.0 2023-11-02T08:28:59.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T08:28:59.584 INFO:teuthology.orchestra.run.smithi116.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-11-02T08:28:59.584 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-11-02T08:28:59.638 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-11-02T08:28:59.639 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-11-02T08:28:59.693 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout b5a9486b4a6f144f83c51b7ec33111a2c86dc081 2023-11-02T08:28:59.749 INFO:tasks.workunit.client.0.smithi116.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-11-02T08:29:42.733 INFO:tasks.workunit.client.0.smithi116.stderr:Note: switching to 'b5a9486b4a6f144f83c51b7ec33111a2c86dc081'. 2023-11-02T08:29:42.734 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-11-02T08:29:42.734 INFO:tasks.workunit.client.0.smithi116.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-11-02T08:29:42.734 INFO:tasks.workunit.client.0.smithi116.stderr:changes and commit them, and you can discard any commits you make in this 2023-11-02T08:29:42.735 INFO:tasks.workunit.client.0.smithi116.stderr:state without impacting any branches by switching back to a branch. 2023-11-02T08:29:42.735 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-11-02T08:29:42.735 INFO:tasks.workunit.client.0.smithi116.stderr:If you want to create a new branch to retain commits you create, you may 2023-11-02T08:29:42.735 INFO:tasks.workunit.client.0.smithi116.stderr:do so (now or later) by using -c with the switch command. Example: 2023-11-02T08:29:42.735 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-11-02T08:29:42.736 INFO:tasks.workunit.client.0.smithi116.stderr: git switch -c 2023-11-02T08:29:42.736 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-11-02T08:29:42.736 INFO:tasks.workunit.client.0.smithi116.stderr:Or undo this operation with: 2023-11-02T08:29:42.736 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-11-02T08:29:42.736 INFO:tasks.workunit.client.0.smithi116.stderr: git switch - 2023-11-02T08:29:42.737 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-11-02T08:29:42.737 INFO:tasks.workunit.client.0.smithi116.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-11-02T08:29:42.737 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-11-02T08:29:42.737 INFO:tasks.workunit.client.0.smithi116.stderr:HEAD is now at b5a9486b4a6 Merge branch 'wip-63387-reef' of https://github.com/ajarr/ceph into wip-yuri6-testing-2023-11-01-0745-reef 2023-11-02T08:29:42.739 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-11-02T08:29:42.794 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-02T08:29:42.794 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-11-02T08:29:42.849 INFO:tasks.workunit:Running workunits matching osd on client.0... 2023-11-02T08:29:42.849 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2023-11-02T08:29:42.850 DEBUG:teuthology.orchestra.run.smithi116:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2023-11-02T08:29:42.915 INFO:tasks.workunit.client.0.smithi116.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-11-02T08:29:42.943 INFO:tasks.workunit.client.0.smithi116.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-11-02T08:29:42.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T08:29:42.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T08:29:42.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2023-11-02T08:29:42.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2023-11-02T08:29:42.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2023-11-02T08:29:42.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2023-11-02T08:29:42.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2023-11-02T08:29:42.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2023-11-02T08:29:42.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/bad-inc-map 2023-11-02T08:29:42.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2023-11-02T08:29:42.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2023-11-02T08:29:42.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:53469 2023-11-02T08:29:42.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:53469 2023-11-02T08:29:42.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2023-11-02T08:29:42.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2023-11-02T08:29:42.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=66c6675f-ccae-48c0-87f7-1cb21dceff29 --auth-supported=none ' 2023-11-02T08:29:42.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:53469 ' 2023-11-02T08:29:42.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2023-11-02T08:29:42.953 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2023-11-02T08:29:42.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-11-02T08:29:42.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2023-11-02T08:29:42.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2023-11-02T08:29:42.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2023-11-02T08:29:42.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2023-11-02T08:29:42.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2023-11-02T08:29:42.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2023-11-02T08:29:42.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T08:29:42.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2023-11-02T08:29:42.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:29:42.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:29:42.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:29:42.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:29:42.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:29:42.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:29:42.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T08:29:42.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T08:29:42.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T08:29:42.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T08:29:42.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T08:29:42.960 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T08:29:42.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:29:42.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T08:29:42.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T08:29:42.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:29:42.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T08:29:42.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T08:29:42.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2023-11-02T08:29:42.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T08:29:42.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:29:42.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79300 2023-11-02T08:29:42.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79300 2023-11-02T08:29:42.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T08:29:42.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T08:29:42.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2023-11-02T08:29:42.966 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T08:29:42.966 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:29:42.967 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79300 2023-11-02T08:29:42.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79300 2023-11-02T08:29:42.967 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T08:29:42.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-11-02T08:29:42.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-11-02T08:29:42.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T08:29:42.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2023-11-02T08:29:42.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2023-11-02T08:29:42.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2023-11-02T08:29:42.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2023-11-02T08:29:42.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2023-11-02T08:29:42.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T08:29:42.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T08:29:42.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T08:29:42.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2023-11-02T08:29:42.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2023-11-02T08:29:43.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T08:29:43.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:29:43.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:29:43.440 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:29:43.440 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:29:43.441 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79300 2023-11-02T08:29:43.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79300/$cluster-$name.asok' 2023-11-02T08:29:43.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.79300/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T08:29:43.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T08:29:43.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T08:29:43.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T08:29:43.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T08:29:43.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T08:29:43.637 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T08:29:43.637 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T08:29:43.637 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T08:29:43.637 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T08:29:43.637 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:29:43.638 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:29:43.638 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79300 2023-11-02T08:29:43.638 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79300/ceph-mon.a.asok 2023-11-02T08:29:43.639 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T08:29:43.639 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79300/ceph-mon.a.asok config get fsid 2023-11-02T08:29:43.723 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T08:29:43.724 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T08:29:43.724 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T08:29:43.724 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T08:29:43.724 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T08:29:43.725 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T08:29:43.725 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T08:29:43.725 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T08:29:43.725 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:29:43.725 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:29:43.726 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79300 2023-11-02T08:29:43.726 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79300/ceph-mon.a.asok 2023-11-02T08:29:43.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T08:29:43.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79300/ceph-mon.a.asok config get mon_host 2023-11-02T08:29:43.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2023-11-02T08:29:43.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/bad-inc-map 2023-11-02T08:29:43.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T08:29:43.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T08:29:43.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T08:29:43.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/bad-inc-map/x 2023-11-02T08:29:43.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T08:29:44.645 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T08:29:44.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:29:44.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:29:44.646 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:29:44.646 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:29:44.647 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79300 2023-11-02T08:29:44.647 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79300/$cluster-$name.asok' 2023-11-02T08:29:44.647 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T08:29:44.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.79300/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T08:29:44.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2023-11-02T08:29:44.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2023-11-02T08:29:44.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:29:44.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T08:29:44.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:29:44.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/0 2023-11-02T08:29:44.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=66c6675f-ccae-48c0-87f7-1cb21dceff29 --auth-supported=none --mon-host=127.0.0.1:53469 ' 2023-11-02T08:29:44.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:29:44.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:29:44.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:29:44.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2023-11-02T08:29:44.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2023-11-02T08:29:44.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:29:44.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:29:44.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2023-11-02T08:29:44.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:29:44.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:29:44.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:29:44.678 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:29:44.678 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:29:44.678 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79300 2023-11-02T08:29:44.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79300/$cluster-$name.asok' 2023-11-02T08:29:44.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79300/$cluster-$name.asok' 2023-11-02T08:29:44.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:29:44.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:29:44.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:29:44.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2023-11-02T08:29:44.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2023-11-02T08:29:44.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:29:44.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:29:44.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:29:44.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:29:44.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:29:44.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:29:44.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/0 2023-11-02T08:29:44.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:29:44.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b4891a1f-55e8-4fd6-a604-ef65af08fab7 2023-11-02T08:29:44.682 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 b4891a1f-55e8-4fd6-a604-ef65af08fab7 2023-11-02T08:29:44.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b4891a1f-55e8-4fd6-a604-ef65af08fab7' 2023-11-02T08:29:44.683 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:29:44.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD4XUNl9i/vKBAA1g2pMmx7/PzSP8Ubdog3vg== 2023-11-02T08:29:44.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD4XUNl9i/vKBAA1g2pMmx7/PzSP8Ubdog3vg=="}' 2023-11-02T08:29:44.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b4891a1f-55e8-4fd6-a604-ef65af08fab7 -i td/bad-inc-map/0/new.json 2023-11-02T08:29:44.999 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:29:45.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/0/new.json 2023-11-02T08:29:45.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=66c6675f-ccae-48c0-87f7-1cb21dceff29 --auth-supported=none --mon-host=127.0.0.1:53469 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.79300/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD4XUNl9i/vKBAA1g2pMmx7/PzSP8Ubdog3vg== --osd-uuid b4891a1f-55e8-4fd6-a604-ef65af08fab7 2023-11-02T08:29:45.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:29:45.074+0000 7fb08f21c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:29:45.079 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:29:45.075+0000 7fb08f21c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:29:45.080 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:29:45.076+0000 7fb08f21c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:29:45.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:29:45.077+0000 7fb08f21c500 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2023-11-02T08:29:45.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:29:45.077+0000 7fb08f21c500 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2023-11-02T08:29:45.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:29:45.077+0000 7fb08f21c500 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2023-11-02T08:29:45.150 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:29:45.146+0000 7fb08f21c500 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2023-11-02T08:29:50.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/0/keyring 2023-11-02T08:29:50.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:29:50.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T08:29:50.855 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T08:29:50.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:29:51.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T08:29:51.356 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T08:29:51.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=66c6675f-ccae-48c0-87f7-1cb21dceff29 --auth-supported=none --mon-host=127.0.0.1:53469 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.79300/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:29:51.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:29:51.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:29:51.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:29:51.379 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:29:51.376+0000 7efd6ca0e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:29:51.402 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:29:51.399+0000 7efd6ca0e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:29:51.403 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:29:51.401+0000 7efd6ca0e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:29:51.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T08:29:51.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:29:51.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:29:51.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:29:51.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:29:51.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:29:51.764 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:29:51.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:29:51.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:29:51.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:29:51.937 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:29:51.934+0000 7efd6ca0e500 -1 Falling back to public interface 2023-11-02T08:29:52.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:29:53.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:29:53.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:29:53.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:29:53.160 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:29:53.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:29:53.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:29:53.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:29:54.188 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:29:54.186+0000 7efd6ca0e500 -1 osd.0 0 log_to_monitors true 2023-11-02T08:29:54.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:29:54.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:29:54.532 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:29:54.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:29:54.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:29:54.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:29:55.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:29:56.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:29:56.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:29:56.092 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:29:56.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:29:56.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:29:56.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:29:56.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:29:57.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:29:57.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:29:57.470 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:29:57.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:29:57.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:29:57.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:29:57.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:29:58.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:29:58.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:29:58.978 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:29:58.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:29:58.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:29:58.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:29:59.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:00.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:00.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:00.361 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:30:00.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:30:00.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:00.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:30:00.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:01.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:01.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:01.874 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:30:01.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:30:01.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:01.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:30:02.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:03.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:03.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:03.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:30:03.261 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:30:03.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:03.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:30:03.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:04.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:04.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:04.651 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:30:04.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:30:04.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:04.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:30:05.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:06.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:06.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:06.322 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:30:06.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:30:06.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:06.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:30:06.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:07.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:07.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:07.796 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:30:07.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:30:07.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:07.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:30:08.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:09.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:09.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:09.323 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:30:09.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:30:09.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:09.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:30:09.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:10.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:10.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:10.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:30:10.877 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:30:10.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:10.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:30:11.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:12.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:12.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:12.588 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:30:12.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:30:12.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:12.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:30:13.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:14.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:14.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:14.136 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:30:14.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:30:14.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:14.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:30:15.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:16.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:16.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:16.603 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:30:16.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:30:16.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:16.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:30:16.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:17.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:17.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:17.983 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T08:30:17.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T08:30:17.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:17.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:30:18.382 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/80386,v1:127.0.0.1:6803/80386] [v2:127.0.0.1:6804/80386,v1:127.0.0.1:6805/80386] exists,up b4891a1f-55e8-4fd6-a604-ef65af08fab7 2023-11-02T08:30:18.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:30:18.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:30:18.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:30:18.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2023-11-02T08:30:18.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2023-11-02T08:30:18.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:30:18.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-11-02T08:30:18.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:30:18.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/1 2023-11-02T08:30:18.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=66c6675f-ccae-48c0-87f7-1cb21dceff29 --auth-supported=none --mon-host=127.0.0.1:53469 ' 2023-11-02T08:30:18.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:30:18.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:30:18.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:30:18.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2023-11-02T08:30:18.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2023-11-02T08:30:18.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:30:18.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:30:18.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2023-11-02T08:30:18.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:30:18.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:30:18.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:30:18.390 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:30:18.390 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:30:18.390 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79300 2023-11-02T08:30:18.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79300/$cluster-$name.asok' 2023-11-02T08:30:18.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79300/$cluster-$name.asok' 2023-11-02T08:30:18.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:30:18.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:30:18.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:30:18.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2023-11-02T08:30:18.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2023-11-02T08:30:18.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:30:18.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:30:18.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:30:18.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:30:18.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:30:18.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:30:18.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/1 2023-11-02T08:30:18.393 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:30:18.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9f285349-9bd9-476a-b0dc-bb6dc2ce9004 2023-11-02T08:30:18.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9f285349-9bd9-476a-b0dc-bb6dc2ce9004' 2023-11-02T08:30:18.394 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 9f285349-9bd9-476a-b0dc-bb6dc2ce9004 2023-11-02T08:30:18.394 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:30:18.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAaXkNlEfq6FxAAxVq8BDN7qOUeqmKYZMEMtg== 2023-11-02T08:30:18.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAaXkNlEfq6FxAAxVq8BDN7qOUeqmKYZMEMtg=="}' 2023-11-02T08:30:18.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9f285349-9bd9-476a-b0dc-bb6dc2ce9004 -i td/bad-inc-map/1/new.json 2023-11-02T08:30:18.897 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:30:18.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/1/new.json 2023-11-02T08:30:18.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=66c6675f-ccae-48c0-87f7-1cb21dceff29 --auth-supported=none --mon-host=127.0.0.1:53469 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.79300/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAaXkNlEfq6FxAAxVq8BDN7qOUeqmKYZMEMtg== --osd-uuid 9f285349-9bd9-476a-b0dc-bb6dc2ce9004 2023-11-02T08:30:18.934 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:18.931+0000 7fac40213500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:30:18.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:18.953+0000 7fac40213500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:30:18.958 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:18.954+0000 7fac40213500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:30:18.958 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:18.955+0000 7fac40213500 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2023-11-02T08:30:18.958 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:18.955+0000 7fac40213500 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2023-11-02T08:30:18.959 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:18.955+0000 7fac40213500 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2023-11-02T08:30:19.057 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:19.054+0000 7fac40213500 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2023-11-02T08:30:24.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/1/keyring 2023-11-02T08:30:24.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:30:24.792 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2023-11-02T08:30:24.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-11-02T08:30:24.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:30:25.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-11-02T08:30:25.303 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T08:30:25.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=66c6675f-ccae-48c0-87f7-1cb21dceff29 --auth-supported=none --mon-host=127.0.0.1:53469 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.79300/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:30:25.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:30:25.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:30:25.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:30:25.326 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:25.323+0000 7fbcfe288500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:30:25.346 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:25.343+0000 7fbcfe288500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:30:25.347 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:25.344+0000 7fbcfe288500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:30:25.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-11-02T08:30:25.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:30:25.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:30:25.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:30:25.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:30:25.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:25.753 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:30:25.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:30:25.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:25.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:30:25.875 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:25.872+0000 7fbcfe288500 -1 Falling back to public interface 2023-11-02T08:30:26.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:27.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:27.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:27.144 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:30:27.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:30:27.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:27.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:30:27.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:28.174 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:28.171+0000 7fbcfe288500 -1 osd.1 0 log_to_monitors true 2023-11-02T08:30:28.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:28.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:28.520 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:30:28.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:30:28.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:28.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:30:28.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:29.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:29.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:29.926 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:30:29.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:30:29.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:29.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:30:30.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:31.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:31.360 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:30:31.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:31.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:30:31.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:31.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:30:31.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:32.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:32.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:32.734 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:30:32.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:30:32.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:32.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:30:33.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:34.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:34.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:34.239 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:30:34.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:30:34.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:34.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:30:34.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:35.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:35.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:35.622 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:30:35.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:30:35.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:35.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:30:36.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:37.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:37.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:37.003 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:30:37.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:30:37.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:37.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:30:37.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:38.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:38.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:38.413 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:30:38.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:30:38.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:38.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:30:38.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:39.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:39.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:39.913 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:30:39.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:30:39.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:39.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:30:40.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:41.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:41.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:41.730 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:30:41.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:30:41.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:41.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:30:42.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:43.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:43.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:43.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:30:43.435 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:30:43.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:43.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:30:44.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:45.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:45.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:45.414 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:30:45.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:30:45.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:45.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:30:46.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:47.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:47.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:47.091 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:30:47.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:30:47.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:47.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:30:47.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:48.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:48.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:48.852 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:30:48.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:30:48.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:48.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:30:49.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:50.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:50.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:50.365 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:30:50.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:30:50.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:50.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:30:50.809 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/82163,v1:127.0.0.1:6811/82163] [v2:127.0.0.1:6812/82163,v1:127.0.0.1:6813/82163] exists,up 9f285349-9bd9-476a-b0dc-bb6dc2ce9004 2023-11-02T08:30:50.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:30:50.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:30:50.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:30:50.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2023-11-02T08:30:50.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2023-11-02T08:30:50.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:30:50.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-11-02T08:30:50.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:30:50.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/2 2023-11-02T08:30:50.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=66c6675f-ccae-48c0-87f7-1cb21dceff29 --auth-supported=none --mon-host=127.0.0.1:53469 ' 2023-11-02T08:30:50.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:30:50.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:30:50.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:30:50.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2023-11-02T08:30:50.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2023-11-02T08:30:50.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:30:50.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:30:50.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2023-11-02T08:30:50.818 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:30:50.819 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:30:50.819 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:30:50.819 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:30:50.819 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:30:50.819 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79300 2023-11-02T08:30:50.820 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79300/$cluster-$name.asok' 2023-11-02T08:30:50.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79300/$cluster-$name.asok' 2023-11-02T08:30:50.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:30:50.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:30:50.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:30:50.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2023-11-02T08:30:50.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2023-11-02T08:30:50.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:30:50.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:30:50.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:30:50.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:30:50.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:30:50.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:30:50.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/2 2023-11-02T08:30:50.822 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:30:50.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cc7189dc-e7e8-4b70-8a76-4ad98be633e9 2023-11-02T08:30:50.823 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 cc7189dc-e7e8-4b70-8a76-4ad98be633e9 2023-11-02T08:30:50.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 cc7189dc-e7e8-4b70-8a76-4ad98be633e9' 2023-11-02T08:30:50.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:30:50.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA6XkNlNvc+MRAAcZVCFNzFmIl8wNjWnP5HfQ== 2023-11-02T08:30:50.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA6XkNlNvc+MRAAcZVCFNzFmIl8wNjWnP5HfQ=="}' 2023-11-02T08:30:50.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cc7189dc-e7e8-4b70-8a76-4ad98be633e9 -i td/bad-inc-map/2/new.json 2023-11-02T08:30:51.299 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:30:51.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/2/new.json 2023-11-02T08:30:51.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=66c6675f-ccae-48c0-87f7-1cb21dceff29 --auth-supported=none --mon-host=127.0.0.1:53469 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.79300/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA6XkNlNvc+MRAAcZVCFNzFmIl8wNjWnP5HfQ== --osd-uuid cc7189dc-e7e8-4b70-8a76-4ad98be633e9 2023-11-02T08:30:51.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:51.332+0000 7fe0bfcc0500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:30:51.338 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:51.334+0000 7fe0bfcc0500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:30:51.338 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:51.335+0000 7fe0bfcc0500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:30:51.339 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:51.335+0000 7fe0bfcc0500 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2023-11-02T08:30:51.339 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:51.335+0000 7fe0bfcc0500 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2023-11-02T08:30:51.339 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:51.335+0000 7fe0bfcc0500 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2023-11-02T08:30:51.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:51.396+0000 7fe0bfcc0500 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2023-11-02T08:30:57.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/2/keyring 2023-11-02T08:30:57.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:30:57.142 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2023-11-02T08:30:57.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-11-02T08:30:57.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:30:57.648 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T08:30:57.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-11-02T08:30:57.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=66c6675f-ccae-48c0-87f7-1cb21dceff29 --auth-supported=none --mon-host=127.0.0.1:53469 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.79300/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:30:57.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:30:57.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:30:57.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:30:57.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:57.669+0000 7faa79337500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:30:57.699 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:57.696+0000 7faa79337500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:30:57.700 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:57.697+0000 7faa79337500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:30:58.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-11-02T08:30:58.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:30:58.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T08:30:58.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:30:58.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:30:58.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:58.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:30:58.053 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:30:58.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:58.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:30:58.229 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:30:58.226+0000 7faa79337500 -1 Falling back to public interface 2023-11-02T08:30:58.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:30:59.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:30:59.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:30:59.421 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:30:59.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:30:59.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:30:59.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:30:59.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:00.485 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:31:00.482+0000 7faa79337500 -1 osd.2 0 log_to_monitors true 2023-11-02T08:31:00.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:00.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:00.842 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:31:00.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:31:00.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:00.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:31:01.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:02.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:02.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:02.221 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:31:02.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:31:02.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:02.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:31:02.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:03.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:03.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:03.628 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:31:03.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:31:03.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:03.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:31:04.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:05.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:05.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:05.034 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:31:05.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:31:05.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:05.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:31:05.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:06.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:06.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:06.466 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:31:06.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:31:06.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:06.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:31:06.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:07.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:07.844 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:31:07.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:07.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:31:07.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:07.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:31:08.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:09.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:09.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:09.346 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:31:09.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:31:09.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:09.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:31:09.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:10.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:10.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:10.727 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:31:10.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:31:10.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:10.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:31:11.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:12.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:12.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:12.106 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:31:12.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:31:12.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:12.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:31:12.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:13.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:13.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:13.923 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:31:13.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:31:13.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:13.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:31:14.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:15.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:15.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:15.759 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:31:15.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:31:15.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:15.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:31:16.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:17.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:17.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:17.362 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:31:17.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:31:17.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:17.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:31:18.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:19.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:19.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:19.149 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:31:19.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:31:19.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:19.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:31:19.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:20.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:20.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:20.991 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:31:20.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:31:20.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:20.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:31:21.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:22.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:22.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:22.649 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:31:22.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:31:22.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:22.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:31:23.045 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/83896,v1:127.0.0.1:6819/83896] [v2:127.0.0.1:6820/83896,v1:127.0.0.1:6821/83896] exists,up cc7189dc-e7e8-4b70-8a76-4ad98be633e9 2023-11-02T08:31:23.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:31:23.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:31:23.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:31:23.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2023-11-02T08:31:23.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2023-11-02T08:31:23.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 8 2023-11-02T08:31:24.134 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'foo' already exists 2023-11-02T08:31:24.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T08:31:25.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2023-11-02T08:31:26.173 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 min_size to 1 2023-11-02T08:31:26.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2023-11-02T08:31:27.297 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 min_size to 2 2023-11-02T08:31:27.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2023-11-02T08:31:32.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2023-11-02T08:31:32.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2023-11-02T08:31:32.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:31:32.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:31:32.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:31:32.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:31:32.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2023-11-02T08:31:32.311 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:31:32.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:31:32.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:32.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:31:32.727 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/80386,v1:127.0.0.1:6803/80386] [v2:127.0.0.1:6804/80386,v1:127.0.0.1:6805/80386] exists,up b4891a1f-55e8-4fd6-a604-ef65af08fab7 2023-11-02T08:31:32.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:31:32.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:31:32.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:31:32.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2023-11-02T08:31:32.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2023-11-02T08:31:32.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:31:32.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:31:32.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:31:32.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:31:32.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2023-11-02T08:31:32.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:31:32.732 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:31:32.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:32.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:31:33.136 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/82163,v1:127.0.0.1:6811/82163] [v2:127.0.0.1:6812/82163,v1:127.0.0.1:6813/82163] exists,up 9f285349-9bd9-476a-b0dc-bb6dc2ce9004 2023-11-02T08:31:33.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:31:33.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:31:33.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:31:33.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2023-11-02T08:31:33.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2023-11-02T08:31:33.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:31:33.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T08:31:33.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:31:33.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:31:33.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2023-11-02T08:31:33.139 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:31:33.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:31:33.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:33.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:31:33.546 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/83896,v1:127.0.0.1:6819/83896] [v2:127.0.0.1:6820/83896,v1:127.0.0.1:6821/83896] exists,up cc7189dc-e7e8-4b70-8a76-4ad98be633e9 2023-11-02T08:31:33.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:31:33.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:31:33.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:31:33.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2023-11-02T08:31:33.548 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:24.005+0000 7faa6a2c2640 -1 osd.2 15 handle_osd_map injecting map crc failure 2023-11-02T08:31:33.548 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:24.451+0000 7faa6a2c2640 -1 osd.2 17 handle_osd_map injecting map crc failure 2023-11-02T08:31:33.548 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:24.827+0000 7faa6a2c2640 -1 osd.2 18 handle_osd_map injecting map crc failure 2023-11-02T08:31:33.548 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:25.834+0000 7faa682be640 -1 osd.2 19 handle_osd_map injecting map crc failure 2023-11-02T08:31:33.549 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:26.244+0000 7faa682be640 -1 osd.2 21 handle_osd_map injecting map crc failure 2023-11-02T08:31:33.549 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:26.555+0000 7faa6a2c2640 -1 osd.2 22 handle_osd_map injecting map crc failure 2023-11-02T08:31:33.549 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:27.036+0000 7faa682be640 -1 osd.2 23 handle_osd_map injecting map crc failure 2023-11-02T08:31:33.549 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:27.036+0000 7faa682be640 -1 osd.2 23 handle_osd_map injecting map crc failure 2023-11-02T08:31:33.550 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:27.292+0000 7faa6a2c2640 -1 osd.2 24 handle_osd_map injecting map crc failure 2023-11-02T08:31:33.550 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:27.729+0000 7faa6a2c2640 -1 osd.2 25 handle_osd_map injecting map crc failure 2023-11-02T08:31:33.550 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:27.730+0000 7faa6a2c2640 -1 osd.2 25 handle_osd_map injecting map crc failure 2023-11-02T08:31:33.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2023-11-02T08:31:33.552 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:24.006+0000 7faa6a2c2640 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2023-11-02T08:31:33.552 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:24.451+0000 7faa6a2c2640 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2023-11-02T08:31:33.552 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:24.827+0000 7faa6a2c2640 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2023-11-02T08:31:33.552 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:25.834+0000 7faa682be640 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2023-11-02T08:31:33.552 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:26.244+0000 7faa682be640 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2023-11-02T08:31:33.553 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:26.555+0000 7faa6a2c2640 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2023-11-02T08:31:33.553 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:27.036+0000 7faa682be640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2023-11-02T08:31:33.553 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:27.037+0000 7faa682be640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2023-11-02T08:31:33.553 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:27.292+0000 7faa6a2c2640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2023-11-02T08:31:33.554 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:27.729+0000 7faa6a2c2640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2023-11-02T08:31:33.554 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:31:27.731+0000 7faa6a2c2640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2023-11-02T08:31:33.554 INFO:tasks.workunit.client.0.smithi116.stdout:success 2023-11-02T08:31:33.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2023-11-02T08:31:33.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2023-11-02T08:31:33.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=foo 2023-11-02T08:31:33.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2023-11-02T08:31:34.058 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'foo' does not exist 2023-11-02T08:31:34.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2023-11-02T08:31:34.070 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:31:34.070 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:31:34.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:31:34.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:31:34.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:31:35.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:31:35.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2023-11-02T08:31:35.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2023-11-02T08:31:35.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T08:31:35.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2023-11-02T08:31:35.385 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:31:35.385 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:31:35.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:31:35.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:31:35.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:31:35.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:31:35.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T08:31:35.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T08:31:35.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T08:31:35.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T08:31:35.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T08:31:35.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T08:31:35.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:31:35.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T08:31:35.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T08:31:35.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:31:35.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T08:31:35.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T08:31:35.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2023-11-02T08:31:35.403 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T08:31:35.403 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:31:35.404 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79300 2023-11-02T08:31:35.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79300 2023-11-02T08:31:35.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T08:31:35.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T08:31:35.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2023-11-02T08:31:35.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/bad-inc-map 0 2023-11-02T08:31:35.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2023-11-02T08:31:35.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-11-02T08:31:35.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2023-11-02T08:31:35.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:31:35.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:31:35.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:31:35.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:31:35.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:31:35.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:31:35.409 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T08:31:35.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T08:31:35.409 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T08:31:35.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T08:31:35.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T08:31:35.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T08:31:35.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:31:35.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T08:31:35.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T08:31:35.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:31:35.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T08:31:35.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-11-02T08:31:35.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2023-11-02T08:31:35.415 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T08:31:35.416 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:31:35.416 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79300 2023-11-02T08:31:35.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79300 2023-11-02T08:31:35.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T08:31:35.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T08:31:35.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2023-11-02T08:31:35.417 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-02T08:31:35.418 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-02T08:31:35.487 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2023-11-02T08:31:35.488 DEBUG:teuthology.orchestra.run.smithi116:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2023-11-02T08:31:35.550 INFO:tasks.workunit.client.0.smithi116.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-11-02T08:31:35.556 INFO:tasks.workunit.client.0.smithi116.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-11-02T08:31:35.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T08:31:35.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T08:31:35.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2023-11-02T08:31:35.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2023-11-02T08:31:35.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2023-11-02T08:31:35.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2023-11-02T08:31:35.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2023-11-02T08:31:35.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2023-11-02T08:31:35.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/divergent-priors 2023-11-02T08:31:35.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2023-11-02T08:31:35.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2023-11-02T08:31:35.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2023-11-02T08:31:35.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2023-11-02T08:31:35.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2023-11-02T08:31:35.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2023-11-02T08:31:35.560 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2023-11-02T08:31:35.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2023-11-02T08:31:35.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2023-11-02T08:31:35.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2023-11-02T08:31:35.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2023-11-02T08:31:35.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2023-11-02T08:31:35.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2023-11-02T08:31:35.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2023-11-02T08:31:35.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2023-11-02T08:31:35.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2023-11-02T08:31:35.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2023-11-02T08:31:35.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2023-11-02T08:31:35.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2023-11-02T08:31:35.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2023-11-02T08:31:35.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2023-11-02T08:31:35.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2023-11-02T08:31:35.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2023-11-02T08:31:35.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2023-11-02T08:31:35.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2023-11-02T08:31:35.566 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2023-11-02T08:31:35.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none ' 2023-11-02T08:31:35.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2023-11-02T08:31:35.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2023-11-02T08:31:35.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2023-11-02T08:31:35.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:31:35.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2023-11-02T08:31:35.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-11-02T08:31:35.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2023-11-02T08:31:35.570 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_divergent_2 2023-11-02T08:31:35.570 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_divergent_3 2023-11-02T08:31:35.570 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_divergent_ec' 2023-11-02T08:31:35.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2023-11-02T08:31:35.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2023-11-02T08:31:35.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2023-11-02T08:31:35.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2023-11-02T08:31:35.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2023-11-02T08:31:35.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T08:31:35.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2023-11-02T08:31:35.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:31:35.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:31:35.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:31:35.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:31:35.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:31:35.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:31:35.573 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T08:31:35.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T08:31:35.574 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T08:31:35.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T08:31:35.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T08:31:35.575 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T08:31:35.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:31:35.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T08:31:35.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T08:31:35.575 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:31:35.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T08:31:35.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T08:31:35.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2023-11-02T08:31:35.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T08:31:35.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:31:35.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:31:35.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.85082 2023-11-02T08:31:35.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T08:31:35.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T08:31:35.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2023-11-02T08:31:35.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T08:31:35.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:31:35.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:31:35.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.85082 2023-11-02T08:31:35.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T08:31:35.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-11-02T08:31:35.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-11-02T08:31:35.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T08:31:35.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2023-11-02T08:31:35.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2023-11-02T08:31:35.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2023-11-02T08:31:35.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2023-11-02T08:31:35.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2023-11-02T08:31:35.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2023-11-02T08:31:35.583 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2023-11-02T08:31:35.583 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2023-11-02T08:31:35.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2023-11-02T08:31:35.594 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:31:35.594 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:31:35.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2023-11-02T08:31:35.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2023-11-02T08:31:35.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T08:31:35.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T08:31:35.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T08:31:35.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2023-11-02T08:31:35.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2023-11-02T08:31:35.953 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T08:31:35.953 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:31:35.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:31:35.954 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:31:35.954 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:31:35.954 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:31:35.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:31:35.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T08:31:36.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T08:31:36.139 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T08:31:36.139 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T08:31:36.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T08:31:36.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T08:31:36.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T08:31:36.140 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T08:31:36.140 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T08:31:36.141 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T08:31:36.141 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:31:36.141 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:31:36.141 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:31:36.141 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-mon.a.asok 2023-11-02T08:31:36.142 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T08:31:36.142 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.85082/ceph-mon.a.asok config get fsid 2023-11-02T08:31:36.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T08:31:36.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T08:31:36.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T08:31:36.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T08:31:36.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T08:31:36.221 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T08:31:36.221 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T08:31:36.221 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T08:31:36.222 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:31:36.222 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:31:36.222 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:31:36.222 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-mon.a.asok 2023-11-02T08:31:36.222 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T08:31:36.223 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.85082/ceph-mon.a.asok config get mon_host 2023-11-02T08:31:36.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2023-11-02T08:31:36.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2023-11-02T08:31:36.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T08:31:36.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T08:31:36.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T08:31:36.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2023-11-02T08:31:36.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T08:31:37.142 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T08:31:37.142 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:31:37.143 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:31:37.143 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:31:37.143 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:31:37.143 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:31:37.143 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:31:37.144 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T08:31:37.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T08:31:37.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2023-11-02T08:31:37.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2023-11-02T08:31:37.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2023-11-02T08:31:37.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:31:37.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T08:31:37.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:31:37.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2023-11-02T08:31:37.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:31:37.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:31:37.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:31:37.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:31:37.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2023-11-02T08:31:37.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2023-11-02T08:31:37.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:31:37.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:31:37.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:31:37.186 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:31:37.186 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:31:37.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:31:37.187 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:31:37.187 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:31:37.188 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:31:37.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:31:37.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:31:37.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:31:37.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:31:37.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:31:37.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:31:37.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:31:37.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:31:37.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:31:37.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:31:37.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:31:37.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:31:37.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:31:37.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2023-11-02T08:31:37.194 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:31:37.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c042b72e-f368-4127-8669-1a40dd6d2fba 2023-11-02T08:31:37.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c042b72e-f368-4127-8669-1a40dd6d2fba' 2023-11-02T08:31:37.196 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 c042b72e-f368-4127-8669-1a40dd6d2fba 2023-11-02T08:31:37.197 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:31:37.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBpXkNlV4vDCxAAxW6wNocleJ1hc+F4v77p7Q== 2023-11-02T08:31:37.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBpXkNlV4vDCxAAxW6wNocleJ1hc+F4v77p7Q=="}' 2023-11-02T08:31:37.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c042b72e-f368-4127-8669-1a40dd6d2fba -i td/divergent-priors/0/new.json 2023-11-02T08:31:37.488 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:31:37.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2023-11-02T08:31:37.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBpXkNlV4vDCxAAxW6wNocleJ1hc+F4v77p7Q== --osd-uuid c042b72e-f368-4127-8669-1a40dd6d2fba 2023-11-02T08:31:37.534 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:31:37.531+0000 7fc7d0174500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:31:37.566 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:31:37.561+0000 7fc7d0174500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:31:37.566 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:31:37.562+0000 7fc7d0174500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:31:37.566 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:31:37.563+0000 7fc7d0174500 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2023-11-02T08:31:37.566 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:31:37.563+0000 7fc7d0174500 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2023-11-02T08:31:37.567 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:31:37.563+0000 7fc7d0174500 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2023-11-02T08:31:37.604 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:31:37.601+0000 7fc7d0174500 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2023-11-02T08:31:42.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2023-11-02T08:31:42.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:31:42.798 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T08:31:42.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T08:31:42.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:31:43.320 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T08:31:43.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T08:31:43.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:31:43.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:31:43.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:31:43.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:31:43.353 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:31:43.350+0000 7f5fade0e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:31:43.355 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:31:43.352+0000 7f5fade0e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:31:43.356 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:31:43.353+0000 7f5fade0e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:31:43.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T08:31:43.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:31:43.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:31:43.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:31:43.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:31:43.741 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:31:43.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:43.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:31:43.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:43.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:31:43.875 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:31:43.873+0000 7f5fade0e500 -1 Falling back to public interface 2023-11-02T08:31:44.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:45.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:45.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:45.144 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:31:45.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:31:45.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:45.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:31:45.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:46.138 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:31:46.135+0000 7f5fade0e500 -1 osd.0 0 log_to_monitors true 2023-11-02T08:31:46.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:46.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:46.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:31:46.570 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:31:46.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:46.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:31:47.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:48.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:48.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:48.085 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:31:48.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:31:48.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:48.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:31:48.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:49.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:49.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:49.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:31:49.497 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:31:49.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:49.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:31:49.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:50.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:50.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:50.989 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:31:50.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:31:50.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:50.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:31:51.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:52.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:52.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:52.392 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:31:52.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:31:52.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:52.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:31:52.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:53.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:53.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:53.867 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:31:53.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:31:53.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:53.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:31:54.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:55.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:55.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:55.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:31:55.255 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:31:55.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:55.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:31:55.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:56.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:56.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:56.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:31:56.641 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:31:56.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:56.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:31:57.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:58.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:58.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:58.206 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:31:58.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:31:58.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:58.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:31:58.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:31:59.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:31:59.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:31:59.767 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:31:59.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:31:59.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:31:59.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:32:00.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:01.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:01.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:01.167 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:32:01.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:32:01.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:01.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:32:02.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:03.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:03.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:03.085 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:32:03.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:32:03.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:03.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:32:03.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:04.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:04.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:04.898 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:32:04.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:32:04.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:04.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:32:05.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:06.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:06.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:06.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:32:06.906 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:32:06.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:06.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:32:07.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:08.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:08.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:08.525 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:32:08.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:32:08.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:08.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:32:08.937 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/86172,v1:127.0.0.1:6803/86172] [v2:127.0.0.1:6804/86172,v1:127.0.0.1:6805/86172] exists,up c042b72e-f368-4127-8669-1a40dd6d2fba 2023-11-02T08:32:08.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:32:08.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:32:08.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:32:08.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2023-11-02T08:32:08.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2023-11-02T08:32:08.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2023-11-02T08:32:08.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:32:08.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-11-02T08:32:08.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:32:08.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2023-11-02T08:32:08.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:32:08.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:32:08.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:32:08.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:32:08.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2023-11-02T08:32:08.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2023-11-02T08:32:08.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:32:08.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:32:08.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:32:08.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:32:08.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:32:08.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:32:08.948 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:32:08.949 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:32:08.949 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:32:08.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:32:08.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:32:08.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:32:08.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:32:08.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:32:08.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:32:08.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:32:08.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:32:08.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:32:08.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:32:08.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:32:08.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:32:08.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:32:08.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2023-11-02T08:32:08.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:32:08.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4ae6c215-0e13-4808-b2e7-66467c5e6ec3 2023-11-02T08:32:08.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4ae6c215-0e13-4808-b2e7-66467c5e6ec3' 2023-11-02T08:32:08.953 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 4ae6c215-0e13-4808-b2e7-66467c5e6ec3 2023-11-02T08:32:08.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:32:08.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCIXkNlGb03ORAATbXJuYVAYCTcpIMvfDyqZg== 2023-11-02T08:32:08.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCIXkNlGb03ORAATbXJuYVAYCTcpIMvfDyqZg=="}' 2023-11-02T08:32:08.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4ae6c215-0e13-4808-b2e7-66467c5e6ec3 -i td/divergent-priors/1/new.json 2023-11-02T08:32:09.538 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:32:09.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2023-11-02T08:32:09.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCIXkNlGb03ORAATbXJuYVAYCTcpIMvfDyqZg== --osd-uuid 4ae6c215-0e13-4808-b2e7-66467c5e6ec3 2023-11-02T08:32:09.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:09.581+0000 7fbbcfe65500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:32:09.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:09.583+0000 7fbbcfe65500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:32:09.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:09.584+0000 7fbbcfe65500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:32:09.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:09.584+0000 7fbbcfe65500 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2023-11-02T08:32:09.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:09.584+0000 7fbbcfe65500 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2023-11-02T08:32:09.588 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:09.584+0000 7fbbcfe65500 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2023-11-02T08:32:09.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:09.659+0000 7fbbcfe65500 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2023-11-02T08:32:15.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2023-11-02T08:32:15.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:32:15.099 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2023-11-02T08:32:15.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-11-02T08:32:15.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:32:15.633 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T08:32:15.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-11-02T08:32:15.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:32:15.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:32:15.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:32:15.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:32:15.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:15.664+0000 7f3f9cc11500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:32:15.675 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:15.672+0000 7f3f9cc11500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:32:15.676 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:15.673+0000 7f3f9cc11500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:32:16.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-11-02T08:32:16.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:32:16.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:32:16.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:32:16.072 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:32:16.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:32:16.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:16.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:32:16.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:16.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:32:16.205 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:16.202+0000 7f3f9cc11500 -1 Falling back to public interface 2023-11-02T08:32:16.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:17.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:17.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:17.491 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:32:17.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:32:17.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:17.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:32:17.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:18.552 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:18.549+0000 7f3f9cc11500 -1 osd.1 0 log_to_monitors true 2023-11-02T08:32:18.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:18.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:18.910 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:32:18.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:32:18.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:18.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:32:19.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:20.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:20.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:20.407 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:32:20.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:32:20.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:20.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:32:20.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:21.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:21.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:21.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:32:21.793 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:32:21.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:21.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:32:22.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:23.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:23.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:23.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:32:23.360 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:32:23.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:23.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:32:23.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:24.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:24.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:24.744 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:32:24.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:32:24.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:24.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:32:25.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:26.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:26.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:26.207 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:32:26.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:32:26.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:26.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:32:26.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:27.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:27.603 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:32:27.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:27.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:32:27.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:27.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:32:28.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:29.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:29.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:29.018 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:32:29.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:32:29.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:29.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:32:29.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:30.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:30.568 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:32:30.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:30.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:32:30.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:30.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:32:31.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:32.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:32.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:32.170 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:32:32.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:32:32.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:32.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:32:32.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:33.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:33.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:33.927 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:32:33.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:32:33.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:33.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:32:34.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:35.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:35.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:35.487 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:32:35.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:32:35.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:35.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:32:35.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:36.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:36.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:36.973 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:32:36.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:32:36.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:36.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:32:38.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:39.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:39.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:39.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:32:39.057 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:32:39.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:39.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:32:39.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:40.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:40.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:40.953 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:32:40.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:32:40.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:40.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:32:41.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:42.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:42.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:42.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T08:32:42.418 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T08:32:42.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:42.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:32:42.791 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/87885,v1:127.0.0.1:6811/87885] [v2:127.0.0.1:6812/87885,v1:127.0.0.1:6813/87885] exists,up 4ae6c215-0e13-4808-b2e7-66467c5e6ec3 2023-11-02T08:32:42.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:32:42.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:32:42.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:32:42.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2023-11-02T08:32:42.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2023-11-02T08:32:42.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2023-11-02T08:32:42.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:32:42.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-11-02T08:32:42.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:32:42.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2023-11-02T08:32:42.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:32:42.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:32:42.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:32:42.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:32:42.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2023-11-02T08:32:42.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2023-11-02T08:32:42.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:32:42.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:32:42.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:32:42.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:32:42.800 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:32:42.800 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:32:42.800 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:32:42.804 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:32:42.805 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:32:42.805 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:32:42.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:32:42.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:32:42.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:32:42.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:32:42.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:32:42.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:32:42.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:32:42.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:32:42.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:32:42.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:32:42.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:32:42.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:32:42.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2023-11-02T08:32:42.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:32:42.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f289b1b9-b765-4d2c-a617-39df3b43faae 2023-11-02T08:32:42.809 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 f289b1b9-b765-4d2c-a617-39df3b43faae 2023-11-02T08:32:42.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f289b1b9-b765-4d2c-a617-39df3b43faae' 2023-11-02T08:32:42.809 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:32:42.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCqXkNl85uUMBAArYvjOoJ+Yhcl0RQ56qM9zA== 2023-11-02T08:32:42.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCqXkNl85uUMBAArYvjOoJ+Yhcl0RQ56qM9zA=="}' 2023-11-02T08:32:42.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f289b1b9-b765-4d2c-a617-39df3b43faae -i td/divergent-priors/2/new.json 2023-11-02T08:32:43.334 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:32:43.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2023-11-02T08:32:43.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCqXkNl85uUMBAArYvjOoJ+Yhcl0RQ56qM9zA== --osd-uuid f289b1b9-b765-4d2c-a617-39df3b43faae 2023-11-02T08:32:43.381 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:43.378+0000 7f98aac0e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:32:43.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:43.380+0000 7f98aac0e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:32:43.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:43.381+0000 7f98aac0e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:32:43.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:43.381+0000 7f98aac0e500 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2023-11-02T08:32:43.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:43.381+0000 7f98aac0e500 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2023-11-02T08:32:43.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:43.381+0000 7f98aac0e500 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2023-11-02T08:32:43.458 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:43.456+0000 7f98aac0e500 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2023-11-02T08:32:48.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2023-11-02T08:32:48.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:32:48.946 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2023-11-02T08:32:48.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-11-02T08:32:48.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:32:49.552 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T08:32:49.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-11-02T08:32:49.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:32:49.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:32:49.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:32:49.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:32:49.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:49.582+0000 7f80fbb92500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:32:49.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:49.584+0000 7f80fbb92500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:32:49.589 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:49.585+0000 7f80fbb92500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:32:50.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-11-02T08:32:50.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:32:50.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T08:32:50.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:32:50.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:32:50.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:50.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:32:50.004 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:32:50.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:50.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:32:50.110 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:50.108+0000 7f80fbb92500 -1 Falling back to public interface 2023-11-02T08:32:50.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:51.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:51.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:51.421 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:32:51.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:32:51.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:51.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:32:51.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:52.416 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:32:52.413+0000 7f80fbb92500 -1 osd.2 0 log_to_monitors true 2023-11-02T08:32:52.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:52.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:52.839 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:32:52.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:32:52.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:52.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:32:53.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:54.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:54.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:54.421 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:32:54.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:32:54.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:54.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:32:54.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:55.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:55.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:55.809 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:32:55.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:32:55.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:55.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:32:56.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:57.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:57.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:57.308 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:32:57.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:32:57.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:57.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:32:57.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:32:58.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:32:58.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:32:58.714 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:32:58.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:32:58.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:32:58.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:32:59.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:33:00.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:33:00.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:33:00.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:33:00.195 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:33:00.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:33:00.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:33:00.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:33:01.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:33:01.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:33:01.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:33:01.584 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:33:01.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:33:01.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:33:02.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:33:03.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:33:03.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:33:03.134 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:33:03.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:33:03.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:33:03.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:33:03.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:33:04.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:33:04.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:33:04.622 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:33:04.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:33:04.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:33:04.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:33:05.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:33:06.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:33:06.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:33:06.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:33:06.268 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:33:06.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:33:06.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:33:07.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:33:08.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:33:08.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:33:08.209 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:33:08.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:33:08.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:33:08.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:33:08.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:33:09.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:33:09.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:33:09.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:33:09.899 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:33:09.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:33:09.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:33:10.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:33:11.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:33:11.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:33:11.416 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:33:11.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:33:11.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:33:11.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:33:12.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:33:13.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:33:13.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:33:13.230 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:33:13.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:33:13.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:33:13.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:33:14.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:33:15.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:33:15.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:33:15.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:33:15.067 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:33:15.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:33:15.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:33:15.497 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/89623,v1:127.0.0.1:6819/89623] [v2:127.0.0.1:6820/89623,v1:127.0.0.1:6821/89623] exists,up f289b1b9-b765-4d2c-a617-39df3b43faae 2023-11-02T08:33:15.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:33:15.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:33:15.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:33:15.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2023-11-02T08:33:16.304 INFO:tasks.workunit.client.0.smithi116.stderr:noout is set 2023-11-02T08:33:16.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2023-11-02T08:33:17.014 INFO:tasks.workunit.client.0.smithi116.stderr:noin is set 2023-11-02T08:33:17.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2023-11-02T08:33:17.876 INFO:tasks.workunit.client.0.smithi116.stderr:nodown is set 2023-11-02T08:33:17.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2023-11-02T08:33:17.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2023-11-02T08:33:18.640 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test' already exists 2023-11-02T08:33:18.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T08:33:19.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2023-11-02T08:33:20.313 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 size to 3 2023-11-02T08:33:20.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2023-11-02T08:33:21.100 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 min_size to 2 2023-11-02T08:33:21.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2023-11-02T08:33:21.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:33:21.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:33:21.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:33:21.529 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:33:21.529 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:33:21.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:33:21.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:33:21.530 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:33:21.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836499 2023-11-02T08:33:21.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836499 2023-11-02T08:33:21.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836499' 2023-11-02T08:33:21.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:33:21.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:33:21.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672973 2023-11-02T08:33:21.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672973 2023-11-02T08:33:21.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836499 1-42949672973' 2023-11-02T08:33:21.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:33:21.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:33:21.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509446 2023-11-02T08:33:21.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509446 2023-11-02T08:33:21.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836499 1-42949672973 2-64424509446' 2023-11-02T08:33:21.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:33:21.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836499 2023-11-02T08:33:21.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:33:21.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:33:21.889 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836499 2023-11-02T08:33:21.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:33:21.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836499 2023-11-02T08:33:21.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2023-11-02T08:33:21.891 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836499 2023-11-02T08:33:21.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:33:22.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836497 -lt 21474836499 2023-11-02T08:33:22.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T08:33:23.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T08:33:23.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:33:23.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836499 -lt 21474836499 2023-11-02T08:33:23.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:33:23.688 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672973 2023-11-02T08:33:23.688 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:33:23.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T08:33:23.689 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672973 2023-11-02T08:33:23.690 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:33:23.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672973 2023-11-02T08:33:23.691 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672973 2023-11-02T08:33:23.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2023-11-02T08:33:23.691 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T08:33:24.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672974 -lt 42949672973 2023-11-02T08:33:24.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:33:24.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509446 2023-11-02T08:33:24.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:33:24.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:33:24.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509446 2023-11-02T08:33:24.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:33:24.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509446 2023-11-02T08:33:24.114 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509446 2023-11-02T08:33:24.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2023-11-02T08:33:24.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:33:24.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509446 -lt 64424509446 2023-11-02T08:33:24.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2023-11-02T08:33:24.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T08:33:24.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T08:33:24.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T08:33:24.494 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-11-02T08:33:24.494 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T08:33:24.494 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T08:33:24.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T08:33:24.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T08:33:24.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T08:33:24.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-11-02T08:33:24.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T08:33:24.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T08:33:24.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T08:33:24.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:33:24.578 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:33:24.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:33:24.962 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:33:24.962 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:33:24.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:33:24.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:33:24.964 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:33:25.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836502 2023-11-02T08:33:25.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836502 2023-11-02T08:33:25.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836502' 2023-11-02T08:33:25.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:33:25.080 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:33:25.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672976 2023-11-02T08:33:25.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672976 2023-11-02T08:33:25.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836502 1-42949672976' 2023-11-02T08:33:25.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:33:25.223 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:33:25.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509448 2023-11-02T08:33:25.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509448 2023-11-02T08:33:25.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836502 1-42949672976 2-64424509448' 2023-11-02T08:33:25.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:33:25.366 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836502 2023-11-02T08:33:25.366 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:33:25.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:33:25.367 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836502 2023-11-02T08:33:25.367 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:33:25.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836502 2023-11-02T08:33:25.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2023-11-02T08:33:25.369 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836502 2023-11-02T08:33:25.369 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:33:25.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836500 -lt 21474836502 2023-11-02T08:33:25.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T08:33:26.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T08:33:26.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:33:27.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836502 -lt 21474836502 2023-11-02T08:33:27.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:33:27.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672976 2023-11-02T08:33:27.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:33:27.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T08:33:27.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672976 2023-11-02T08:33:27.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:33:27.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672976 2023-11-02T08:33:27.132 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672976 2023-11-02T08:33:27.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2023-11-02T08:33:27.133 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T08:33:27.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672977 -lt 42949672976 2023-11-02T08:33:27.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:33:27.552 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509448 2023-11-02T08:33:27.552 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:33:27.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:33:27.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509448 2023-11-02T08:33:27.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:33:27.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509448 2023-11-02T08:33:27.554 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509448 2023-11-02T08:33:27.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2023-11-02T08:33:27.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:33:27.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509449 -lt 64424509448 2023-11-02T08:33:27.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T08:33:27.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:33:27.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:33:28.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2023-11-02T08:33:28.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:33:28.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:33:28.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:33:28.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:33:28.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:33:28.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:33:28.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:33:28.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T08:33:28.805 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:33:28.805 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:33:28.805 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:33:29.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2023-11-02T08:33:29.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T08:33:29.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T08:33:29.310 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2023-11-02T08:33:29.311 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2023-11-02T08:33:29.699 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:33:29.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2023-11-02T08:33:29.710 INFO:tasks.workunit.client.0.smithi116.stdout:primary and soon to be divergent is 1 2023-11-02T08:33:29.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2023-11-02T08:33:29.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2023-11-02T08:33:30.063 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:33:30.064 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-02T08:33:19.247780+0000 0'0 28:14 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:33:18.225100+0000 0'0 2023-11-02T08:33:18.225100+0000 0 0 periodic scrub scheduled @ 2023-11-03T15:37:21.640877+0000 0 0 2023-11-02T08:33:30.064 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:33:30.064 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:33:30.065 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:33:30.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2023-11-02T08:33:30.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2023-11-02T08:33:30.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2023-11-02T08:33:30.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2023-11-02T08:33:30.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2023-11-02T08:33:30.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2023-11-02T08:33:30.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2023-11-02T08:33:30.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2023-11-02T08:33:30.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2023-11-02T08:33:30.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2023-11-02T08:33:30.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2023-11-02T08:33:30.077 INFO:tasks.workunit.client.0.smithi116.stdout:writing initial objects 2023-11-02T08:33:30.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2023-11-02T08:33:30.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:30.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2023-11-02T08:33:30.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:30.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2023-11-02T08:33:30.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:30.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2023-11-02T08:33:30.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:30.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2023-11-02T08:33:30.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:30.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2023-11-02T08:33:30.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:30.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2023-11-02T08:33:30.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:30.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2023-11-02T08:33:31.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:31.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2023-11-02T08:33:31.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:31.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2023-11-02T08:33:31.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:31.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2023-11-02T08:33:31.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:31.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2023-11-02T08:33:31.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:31.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2023-11-02T08:33:31.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:31.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2023-11-02T08:33:32.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:32.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2023-11-02T08:33:32.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:32.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2023-11-02T08:33:32.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:32.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2023-11-02T08:33:32.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:32.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2023-11-02T08:33:32.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:32.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2023-11-02T08:33:32.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:32.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2023-11-02T08:33:32.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:32.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2023-11-02T08:33:33.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:33.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2023-11-02T08:33:33.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:33.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2023-11-02T08:33:33.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:33.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2023-11-02T08:33:33.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:33.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2023-11-02T08:33:33.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:33.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2023-11-02T08:33:33.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:33.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2023-11-02T08:33:33.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:33.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2023-11-02T08:33:34.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:34.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2023-11-02T08:33:34.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:34.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2023-11-02T08:33:34.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:34.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2023-11-02T08:33:34.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:34.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2023-11-02T08:33:34.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:34.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2023-11-02T08:33:34.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:34.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2023-11-02T08:33:34.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:34.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2023-11-02T08:33:35.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:35.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2023-11-02T08:33:35.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:35.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2023-11-02T08:33:35.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:35.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2023-11-02T08:33:35.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:35.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2023-11-02T08:33:35.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:35.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2023-11-02T08:33:35.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:35.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2023-11-02T08:33:35.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:35.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2023-11-02T08:33:36.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:36.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2023-11-02T08:33:36.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:36.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2023-11-02T08:33:36.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:36.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2023-11-02T08:33:36.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:36.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2023-11-02T08:33:36.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:36.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2023-11-02T08:33:36.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:36.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2023-11-02T08:33:37.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:37.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2023-11-02T08:33:37.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:37.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2023-11-02T08:33:37.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:37.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2023-11-02T08:33:37.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:37.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2023-11-02T08:33:37.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:37.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2023-11-02T08:33:37.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:37.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2023-11-02T08:33:37.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:37.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2023-11-02T08:33:38.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:38.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2023-11-02T08:33:38.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:38.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2023-11-02T08:33:38.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:38.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2023-11-02T08:33:38.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:38.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2023-11-02T08:33:38.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:38.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2023-11-02T08:33:38.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:38.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2023-11-02T08:33:38.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:38.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2023-11-02T08:33:39.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:39.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2023-11-02T08:33:39.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:39.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2023-11-02T08:33:39.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:39.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2023-11-02T08:33:39.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:39.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2023-11-02T08:33:39.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:39.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2023-11-02T08:33:39.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:39.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2023-11-02T08:33:39.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:39.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2023-11-02T08:33:40.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:40.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2023-11-02T08:33:40.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:40.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2023-11-02T08:33:40.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:40.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2023-11-02T08:33:40.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:40.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2023-11-02T08:33:40.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:40.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2023-11-02T08:33:40.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:40.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2023-11-02T08:33:40.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:40.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2023-11-02T08:33:41.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:41.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2023-11-02T08:33:41.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:41.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2023-11-02T08:33:41.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:41.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2023-11-02T08:33:41.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:41.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2023-11-02T08:33:41.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:41.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2023-11-02T08:33:41.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:41.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2023-11-02T08:33:42.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:42.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2023-11-02T08:33:42.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:42.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2023-11-02T08:33:42.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:42.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2023-11-02T08:33:42.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:42.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2023-11-02T08:33:42.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:42.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2023-11-02T08:33:42.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:42.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2023-11-02T08:33:42.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:42.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2023-11-02T08:33:43.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:43.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2023-11-02T08:33:43.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:43.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2023-11-02T08:33:43.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:43.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2023-11-02T08:33:43.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:43.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2023-11-02T08:33:43.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:43.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2023-11-02T08:33:43.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:43.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2023-11-02T08:33:43.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:43.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2023-11-02T08:33:44.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:44.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2023-11-02T08:33:44.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:44.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2023-11-02T08:33:44.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:44.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2023-11-02T08:33:44.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:44.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2023-11-02T08:33:44.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2023-11-02T08:33:44.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2023-11-02T08:33:44.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2023-11-02T08:33:44.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2023-11-02T08:33:44.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T08:33:44.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T08:33:44.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T08:33:44.809 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2023-11-02T08:33:44.810 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T08:33:44.810 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T08:33:44.810 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T08:33:44.810 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T08:33:44.810 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T08:33:44.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2023-11-02T08:33:44.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T08:33:44.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T08:33:44.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T08:33:44.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:33:44.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:33:45.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:33:45.235 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:33:45.235 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:33:45.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:33:45.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:33:45.236 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:33:45.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836508 2023-11-02T08:33:45.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836508 2023-11-02T08:33:45.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836508' 2023-11-02T08:33:45.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:33:45.354 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:33:45.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672982 2023-11-02T08:33:45.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672982 2023-11-02T08:33:45.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836508 1-42949672982' 2023-11-02T08:33:45.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:33:45.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:33:45.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509454 2023-11-02T08:33:45.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509454 2023-11-02T08:33:45.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836508 1-42949672982 2-64424509454' 2023-11-02T08:33:45.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:33:45.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836508 2023-11-02T08:33:45.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:33:45.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:33:45.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836508 2023-11-02T08:33:45.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:33:45.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836508 2023-11-02T08:33:45.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2023-11-02T08:33:45.593 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836508 2023-11-02T08:33:45.593 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:33:45.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836506 -lt 21474836508 2023-11-02T08:33:45.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T08:33:46.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T08:33:46.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:33:47.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836508 -lt 21474836508 2023-11-02T08:33:47.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:33:47.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672982 2023-11-02T08:33:47.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:33:47.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T08:33:47.385 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672982 2023-11-02T08:33:47.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:33:47.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672982 2023-11-02T08:33:47.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2023-11-02T08:33:47.387 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672982 2023-11-02T08:33:47.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T08:33:47.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672983 -lt 42949672982 2023-11-02T08:33:47.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:33:47.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509454 2023-11-02T08:33:47.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:33:47.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:33:47.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509454 2023-11-02T08:33:47.825 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:33:47.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509454 2023-11-02T08:33:47.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2023-11-02T08:33:47.827 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509454 2023-11-02T08:33:47.827 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:33:48.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509455 -lt 64424509454 2023-11-02T08:33:48.211 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T08:33:48.211 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:33:48.211 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:33:48.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2023-11-02T08:33:48.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:33:48.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:33:48.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:33:48.731 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:33:48.731 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:33:48.731 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:33:48.731 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:33:49.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T08:33:49.120 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:33:49.120 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:33:49.120 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:33:49.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2023-11-02T08:33:49.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T08:33:49.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T08:33:49.653 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2023-11-02T08:33:49.653 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2023-11-02T08:33:49.653 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2023-11-02T08:33:49.654 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2023-11-02T08:33:49.654 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2023-11-02T08:33:50.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2023-11-02T08:33:50.086 INFO:tasks.workunit.client.0.smithi116.stdout:blackholing osds 0 2 2023-11-02T08:33:50.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2023-11-02T08:33:50.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2023-11-02T08:33:50.441 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:33:50.441 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 0 0 0 48300 0 0 16 84 16 active+clean 2023-11-02T08:33:19.247780+0000 29'100 29:114 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:33:18.225100+0000 0'0 2023-11-02T08:33:18.225100+0000 0 0 periodic scrub scheduled @ 2023-11-03T15:37:21.640877+0000 0 0 2023-11-02T08:33:50.441 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:33:50.442 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:33:50.442 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:33:50.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2023-11-02T08:33:50.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2023-11-02T08:33:50.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-11-02T08:33:50.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-11-02T08:33:50.453 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:33:50.453 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:33:50.454 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:33:50.454 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.0.asok 2023-11-02T08:33:50.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2023-11-02T08:33:50.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.0.asok config set objectstore_blackhole 1 2023-11-02T08:33:50.530 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T08:33:50.531 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2023-11-02T08:33:50.531 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T08:33:50.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2023-11-02T08:33:50.537 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2023-11-02T08:33:50.537 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-11-02T08:33:50.537 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-11-02T08:33:50.537 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:33:50.538 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:33:50.538 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:33:50.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.2.asok 2023-11-02T08:33:50.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2023-11-02T08:33:50.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.2.asok config set objectstore_blackhole 1 2023-11-02T08:33:50.612 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T08:33:50.613 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2023-11-02T08:33:50.613 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T08:33:50.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2023-11-02T08:33:50.620 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2023-11-02T08:33:50.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2023-11-02T08:33:50.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2023-11-02T08:33:50.621 INFO:tasks.workunit.client.0.smithi116.stdout:writing divergent object 2023-11-02T08:33:50.621 INFO:tasks.workunit.client.0.smithi116.stdout:create missing divergent object 2023-11-02T08:33:50.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2023-11-02T08:33:50.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2023-11-02T08:33:50.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2023-11-02T08:33:50.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: inject_eio: local pooltype=rep 2023-11-02T08:33:50.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: inject_eio: shift 2023-11-02T08:33:50.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: inject_eio: local which=data 2023-11-02T08:33:50.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: inject_eio: shift 2023-11-02T08:33:50.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: inject_eio: local poolname=test 2023-11-02T08:33:50.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: inject_eio: shift 2023-11-02T08:33:50.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: inject_eio: local objname=existing_99 2023-11-02T08:33:50.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: inject_eio: shift 2023-11-02T08:33:50.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: inject_eio: local dir=td/divergent-priors 2023-11-02T08:33:50.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: inject_eio: shift 2023-11-02T08:33:50.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2359: inject_eio: local shard_id=0 2023-11-02T08:33:50.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2360: inject_eio: shift 2023-11-02T08:33:50.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: get_osds test existing_99 2023-11-02T08:33:50.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test 2023-11-02T08:33:50.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=existing_99 2023-11-02T08:33:50.628 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test existing_99 2023-11-02T08:33:50.628 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2023-11-02T08:33:51.046 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2023-11-02T08:33:51.047 INFO:tasks.workunit.client.0.smithi116.stderr:0 2023-11-02T08:33:51.047 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:33:51.047 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 2023-11-02T08:33:51.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: initial_osds=('1' '0' '2') 2023-11-02T08:33:51.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2362: inject_eio: local -a initial_osds 2023-11-02T08:33:51.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2363: inject_eio: local osd_id=1 2023-11-02T08:33:51.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2364: inject_eio: '[' rep '!=' ec ']' 2023-11-02T08:33:51.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: shard_id= 2023-11-02T08:33:51.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: cat td/divergent-priors/1/type 2023-11-02T08:33:51.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: type=bluestore 2023-11-02T08:33:51.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2023-11-02T08:33:51.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2023-11-02T08:33:51.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2023-11-02T08:33:51.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2023-11-02T08:33:51.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2023-11-02T08:33:51.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2023-11-02T08:33:51.052 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2023-11-02T08:33:51.052 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-11-02T08:33:51.052 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-11-02T08:33:51.052 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:33:51.052 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:33:51.053 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:33:51.053 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.1.asok 2023-11-02T08:33:51.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.85082/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2023-11-02T08:33:51.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2023-11-02T08:33:51.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local loop=0 2023-11-02T08:33:51.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: grep -q Invalid 2023-11-02T08:33:51.135 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: get_asok_path osd.1 2023-11-02T08:33:51.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-11-02T08:33:51.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-11-02T08:33:51.136 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:33:51.136 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:33:51.136 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:33:51.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.1.asok 2023-11-02T08:33:51.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: CEPH_ARGS= 2023-11-02T08:33:51.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.1.asok injectdataerr test existing_99 2023-11-02T08:33:51.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2023-11-02T08:33:51.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2023-11-02T08:34:01.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2023-11-02T08:34:01.241 INFO:tasks.workunit.client.0.smithi116.stderr:rados: no process found 2023-11-02T08:34:01.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2023-11-02T08:34:01.242 INFO:tasks.workunit.client.0.smithi116.stdout:killing all the osds 2023-11-02T08:34:01.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2023-11-02T08:34:01.605 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:34:01.605 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 0 0 0 48300 0 0 17 84 17 active+clean 2023-11-02T08:33:51.475285+0000 29'101 29:123 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:33:18.225100+0000 0'0 2023-11-02T08:33:18.225100+0000 0 0 periodic scrub scheduled @ 2023-11-03T15:37:21.640877+0000 0 0 2023-11-02T08:34:01.606 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:34:01.606 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:34:01.606 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:34:01.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2023-11-02T08:34:01.616 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:34:01.616 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:34:01.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:34:01.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:34:01.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:34:01.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:34:01.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2023-11-02T08:34:01.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2023-11-02T08:34:02.246 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 is already down. 2023-11-02T08:34:02.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2023-11-02T08:34:02.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2023-11-02T08:34:02.795 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 is already down. 2023-11-02T08:34:02.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2023-11-02T08:34:02.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2023-11-02T08:34:03.461 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 is already down. 2023-11-02T08:34:03.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2023-11-02T08:34:03.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2023-11-02T08:34:03.983 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 is already out. 2023-11-02T08:34:03.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2023-11-02T08:34:03.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2023-11-02T08:34:04.543 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 is already out. 2023-11-02T08:34:04.554 INFO:tasks.workunit.client.0.smithi116.stdout:bringing up non_divergent 0 2 2023-11-02T08:34:04.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2023-11-02T08:34:04.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2023-11-02T08:34:04.938 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:34:04.938 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 0 0 0 48300 0 0 17 84 17 stale+active+clean 2023-11-02T08:33:51.475285+0000 29'101 29:123 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:33:18.225100+0000 0'0 2023-11-02T08:33:18.225100+0000 0 0 periodic scrub scheduled @ 2023-11-03T15:37:21.640877+0000 0 0 2023-11-02T08:34:04.938 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:34:04.938 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:34:04.939 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:34:04.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2023-11-02T08:34:04.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2023-11-02T08:34:04.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2023-11-02T08:34:04.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:34:04.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-11-02T08:34:04.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:34:04.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2023-11-02T08:34:04.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:34:04.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:34:04.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:34:04.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:34:04.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2023-11-02T08:34:04.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2023-11-02T08:34:04.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:34:04.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:34:04.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:34:04.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:34:04.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:34:04.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:34:04.955 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:34:04.955 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:34:04.955 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:34:04.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:34:04.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:34:04.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:34:04.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:34:04.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:34:04.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:34:04.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:34:04.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:34:04.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:34:04.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:34:04.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:34:04.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2023-11-02T08:34:04.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-11-02T08:34:04.958 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T08:34:04.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:34:04.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2023-11-02T08:34:04.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-11-02T08:34:04.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:34:04.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:34:04.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:34:04.984 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:34:04.980+0000 7f58f6b74500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:34:04.985 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:34:04.982+0000 7f58f6b74500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:34:04.986 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:34:04.983+0000 7f58f6b74500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:34:05.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-11-02T08:34:05.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:34:05.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:34:05.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:34:05.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:34:05.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:34:05.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:34:05.341 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:34:05.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:34:05.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:34:05.510 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:34:05.507+0000 7f58f6b74500 -1 Falling back to public interface 2023-11-02T08:34:05.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:34:06.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:34:06.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:34:06.771 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:34:06.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:34:06.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:34:06.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:34:07.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:34:08.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:34:08.093+0000 7f58f6b74500 -1 osd.0 29 log_to_monitors true 2023-11-02T08:34:08.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:34:08.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:34:08.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:34:08.158 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:34:08.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:34:08.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:34:08.593 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/93866,v1:127.0.0.1:6803/93866] [v2:127.0.0.1:6804/93866,v1:127.0.0.1:6805/93866] exists,up c042b72e-f368-4127-8669-1a40dd6d2fba 2023-11-02T08:34:08.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:34:08.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:34:08.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:34:08.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2023-11-02T08:34:08.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2023-11-02T08:34:08.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2023-11-02T08:34:08.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:34:08.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2023-11-02T08:34:08.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:34:08.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2023-11-02T08:34:08.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:34:08.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:34:08.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:34:08.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:34:08.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2023-11-02T08:34:08.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2023-11-02T08:34:08.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:34:08.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:34:08.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:34:08.600 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:34:08.600 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:34:08.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:34:08.601 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:34:08.601 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:34:08.601 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:34:08.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:34:08.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:34:08.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:34:08.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:34:08.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:34:08.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:34:08.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:34:08.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:34:08.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:34:08.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:34:08.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:34:08.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2023-11-02T08:34:08.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2023-11-02T08:34:08.604 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T08:34:08.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:34:08.605 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2023-11-02T08:34:08.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2023-11-02T08:34:08.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:34:08.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:34:08.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:34:08.628 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:34:08.625+0000 7f14c540d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:34:08.630 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:34:08.627+0000 7f14c540d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:34:08.631 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:34:08.628+0000 7f14c540d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:34:09.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2023-11-02T08:34:09.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:34:09.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T08:34:09.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:34:09.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:34:09.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:34:09.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:34:09.013 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:34:09.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:34:09.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:34:09.150 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:34:09.147+0000 7f14c540d500 -1 Falling back to public interface 2023-11-02T08:34:09.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:34:10.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:34:10.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:34:10.400 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:34:10.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:34:10.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:34:10.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:34:10.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:34:11.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:34:11.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:34:11.786 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:34:11.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:34:11.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:34:11.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:34:11.829 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:34:11.824+0000 7f14c540d500 -1 osd.2 29 log_to_monitors true 2023-11-02T08:34:12.194 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/94410,v1:127.0.0.1:6811/94410] [v2:127.0.0.1:6812/94410,v1:127.0.0.1:6813/94410] exists,up f289b1b9-b765-4d2c-a617-39df3b43faae 2023-11-02T08:34:12.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:34:12.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:34:12.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:34:12.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2023-11-02T08:34:12.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2023-11-02T08:34:12.771 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 is already in. 2023-11-02T08:34:12.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2023-11-02T08:34:12.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2023-11-02T08:34:13.412 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 is already in. 2023-11-02T08:34:13.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2023-11-02T08:34:13.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2023-11-02T08:34:13.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T08:34:13.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T08:34:13.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T08:34:13.425 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2023-11-02T08:34:13.426 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T08:34:13.426 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T08:34:13.426 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T08:34:13.426 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T08:34:13.426 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T08:34:13.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2023-11-02T08:34:13.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T08:34:13.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T08:34:13.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T08:34:13.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:34:13.474 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:34:13.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:34:13.894 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:34:13.895 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:34:13.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:34:13.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:34:13.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:34:14.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=154618822660 2023-11-02T08:34:14.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 154618822660 2023-11-02T08:34:14.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-154618822660' 2023-11-02T08:34:14.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:34:14.015 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:34:14.099 INFO:tasks.workunit.client.0.smithi116.stderr:Error ENXIO: problem getting command descriptions from osd.1 2023-11-02T08:34:14.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2023-11-02T08:34:14.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2023-11-02T08:34:14.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2023-11-02T08:34:14.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:34:14.102 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:34:14.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=163208757251 2023-11-02T08:34:14.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 163208757251 2023-11-02T08:34:14.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2023-11-02T08:34:14.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:34:14.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-154618822660 2023-11-02T08:34:14.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:34:14.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:34:14.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-154618822660 2023-11-02T08:34:14.222 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:34:14.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=154618822660 2023-11-02T08:34:14.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2023-11-02T08:34:14.223 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 154618822660 2023-11-02T08:34:14.223 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:34:14.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 1 -lt 154618822660 2023-11-02T08:34:14.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T08:34:15.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T08:34:15.647 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:34:16.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 154618822660 -lt 154618822660 2023-11-02T08:34:16.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:34:16.028 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-163208757251 2023-11-02T08:34:16.028 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:34:16.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:34:16.029 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-163208757251 2023-11-02T08:34:16.029 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:34:16.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=163208757251 2023-11-02T08:34:16.030 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 163208757251 2023-11-02T08:34:16.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2023-11-02T08:34:16.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:34:16.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 163208757251 -lt 163208757251 2023-11-02T08:34:16.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T08:34:16.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:34:16.451 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:34:16.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2023-11-02T08:34:16.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:34:16.956 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:34:16.956 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:34:16.956 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:34:16.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:34:16.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:34:16.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:34:17.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:34:17.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:34:17.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:34:17.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:34:17.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:34:17.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2023-11-02T08:34:17.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2023-11-02T08:34:17.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2023-11-02T08:34:17.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:34:17.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T08:34:17.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:34:17.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:34:17.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:34:17.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:34:17.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:34:17.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:34:17.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:34:17.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:34:18.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:34:18.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:34:18.332 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:34:18.332 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:34:18.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:34:18.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:34:18.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:34:18.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:34:18.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:34:18.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:34:18.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:34:18.843 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:34:18.843 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:34:19.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:34:19.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:34:19.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2023-11-02T08:34:19.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:34:19.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2023-11-02T08:34:19.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:34:19.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:34:19.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:34:19.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:34:19.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:34:19.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:34:19.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:34:19.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:34:19.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:34:19.956 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:34:19.956 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:34:19.956 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:34:20.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:34:20.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:34:20.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:34:20.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:34:20.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:34:20.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:34:20.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:34:20.466 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:34:20.466 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:34:20.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:34:20.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:34:20.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2023-11-02T08:34:20.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:34:20.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2023-11-02T08:34:21.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:34:21.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:34:21.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:34:21.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:34:21.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:34:21.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:34:21.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:34:21.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:34:21.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:34:21.751 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:34:21.751 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:34:21.751 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:34:22.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:34:22.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:34:22.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:34:22.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:34:22.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:34:22.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:34:22.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:34:22.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:34:22.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:34:22.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:34:22.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:34:22.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2023-11-02T08:34:22.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:34:22.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2023-11-02T08:34:23.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:34:23.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:34:23.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:34:23.602 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:34:23.602 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:34:23.602 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:34:23.602 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:34:23.602 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:34:23.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:34:23.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:34:23.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:34:23.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:34:24.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:34:24.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:34:24.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:34:24.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:34:24.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:34:24.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:34:24.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:34:24.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:34:24.507 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:34:25.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:34:25.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:34:25.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2023-11-02T08:34:25.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:34:25.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2023-11-02T08:34:26.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:34:26.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:34:26.622 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:34:26.622 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:34:26.622 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:34:26.622 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:34:26.623 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:34:26.623 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:34:27.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:34:27.000 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:34:27.001 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:34:27.001 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:34:27.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:34:27.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:34:27.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:34:27.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:34:27.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:34:27.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:34:27.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:34:27.513 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:34:27.513 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:34:28.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:34:28.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:34:28.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2023-11-02T08:34:28.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:34:28.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2023-11-02T08:34:31.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:34:31.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:34:31.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:34:31.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:34:31.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:34:31.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:34:31.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:34:31.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:34:31.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:34:31.600 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:34:31.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:34:31.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:34:32.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:34:32.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:34:32.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:34:32.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:34:32.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:34:32.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:34:32.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:34:32.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:34:32.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:34:32.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:34:32.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:34:32.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2023-11-02T08:34:32.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:34:32.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2023-11-02T08:34:39.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:34:39.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:34:39.040 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:34:39.040 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:34:39.040 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:34:39.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:34:39.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:34:39.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:34:39.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:34:39.422 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:34:39.422 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:34:39.422 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:34:39.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:34:39.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:34:39.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:34:39.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:34:39.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:34:39.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:34:39.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:34:39.925 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:34:39.925 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:34:40.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:34:40.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:34:40.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2023-11-02T08:34:40.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:34:40.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2023-11-02T08:34:47.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:34:47.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:34:47.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:34:47.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:34:47.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:34:47.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:34:47.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:34:47.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:34:48.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:34:48.144 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:34:48.145 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:34:48.145 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:34:48.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:34:48.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:34:48.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:34:48.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:34:48.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:34:48.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:34:48.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:34:48.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:34:48.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:34:49.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:34:49.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:34:49.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2023-11-02T08:34:49.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2023-11-02T08:34:49.700 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T08:34:49.700 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_fingerprint": "054c1325-1085-4470-8fd9-d340e82fa4a5", 2023-11-02T08:34:49.701 INFO:tasks.workunit.client.0.smithi116.stdout: "version": "18.2.0-1159-gb5a9486b", 2023-11-02T08:34:49.701 INFO:tasks.workunit.client.0.smithi116.stdout: "commit": "b5a9486b4a6f144f83c51b7ec33111a2c86dc081", 2023-11-02T08:34:49.701 INFO:tasks.workunit.client.0.smithi116.stdout: "timestamp": "2023-11-02T08:34:49.696810+0000", 2023-11-02T08:34:49.701 INFO:tasks.workunit.client.0.smithi116.stdout: "tag": "", 2023-11-02T08:34:49.701 INFO:tasks.workunit.client.0.smithi116.stdout: "health": { 2023-11-02T08:34:49.702 INFO:tasks.workunit.client.0.smithi116.stdout: "status": "HEALTH_WARN", 2023-11-02T08:34:49.702 INFO:tasks.workunit.client.0.smithi116.stdout: "checks": { 2023-11-02T08:34:49.702 INFO:tasks.workunit.client.0.smithi116.stdout: "OSDMAP_FLAGS": { 2023-11-02T08:34:49.705 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:34:49.706 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:34:49.706 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "nodown,noout,noin flag(s) set", 2023-11-02T08:34:49.706 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 17 2023-11-02T08:34:49.706 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.706 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [], 2023-11-02T08:34:49.707 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:34:49.707 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.707 INFO:tasks.workunit.client.0.smithi116.stdout: "OSD_DOWN": { 2023-11-02T08:34:49.707 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:34:49.707 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:34:49.707 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "1 osds down", 2023-11-02T08:34:49.708 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 1 2023-11-02T08:34:49.708 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.708 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:34:49.708 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.708 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "osd.1 (root=default,host=smithi116) is down" 2023-11-02T08:34:49.709 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.709 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:34:49.709 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:34:49.709 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.709 INFO:tasks.workunit.client.0.smithi116.stdout: "PG_DEGRADED": { 2023-11-02T08:34:49.709 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:34:49.710 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:34:49.710 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2023-11-02T08:34:49.710 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 1 2023-11-02T08:34:49.710 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.710 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:34:49.711 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.711 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2023-11-02T08:34:49.711 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.711 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:34:49.711 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:34:49.712 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.712 INFO:tasks.workunit.client.0.smithi116.stdout: "POOL_APP_NOT_ENABLED": { 2023-11-02T08:34:49.712 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:34:49.712 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:34:49.712 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "1 pool(s) do not have an application enabled", 2023-11-02T08:34:49.713 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 1 2023-11-02T08:34:49.713 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.713 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:34:49.713 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.713 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "application not enabled on pool 'test'" 2023-11-02T08:34:49.713 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.714 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.714 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2023-11-02T08:34:49.714 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.714 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:34:49.714 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:34:49.715 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.715 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.715 INFO:tasks.workunit.client.0.smithi116.stdout: "mutes": [] 2023-11-02T08:34:49.715 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.716 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap_first_committed": 1, 2023-11-02T08:34:49.716 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap_last_committed": 1, 2023-11-02T08:34:49.717 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap": { 2023-11-02T08:34:49.717 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 1, 2023-11-02T08:34:49.717 INFO:tasks.workunit.client.0.smithi116.stdout: "fsid": "0c9a372f-cec7-47d9-a309-8248453aabeb", 2023-11-02T08:34:49.717 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:31:35.616252Z", 2023-11-02T08:34:49.717 INFO:tasks.workunit.client.0.smithi116.stdout: "created": "2023-11-02T08:31:35.616252Z", 2023-11-02T08:34:49.718 INFO:tasks.workunit.client.0.smithi116.stdout: "min_mon_release": 18, 2023-11-02T08:34:49.718 INFO:tasks.workunit.client.0.smithi116.stdout: "min_mon_release_name": "reef", 2023-11-02T08:34:49.718 INFO:tasks.workunit.client.0.smithi116.stdout: "election_strategy": 1, 2023-11-02T08:34:49.718 INFO:tasks.workunit.client.0.smithi116.stdout: "disallowed_leaders: ": "", 2023-11-02T08:34:49.718 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode": false, 2023-11-02T08:34:49.719 INFO:tasks.workunit.client.0.smithi116.stdout: "tiebreaker_mon": "", 2023-11-02T08:34:49.719 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_ranks: ": "", 2023-11-02T08:34:49.719 INFO:tasks.workunit.client.0.smithi116.stdout: "features": { 2023-11-02T08:34:49.719 INFO:tasks.workunit.client.0.smithi116.stdout: "persistent": [ 2023-11-02T08:34:49.719 INFO:tasks.workunit.client.0.smithi116.stdout: "kraken", 2023-11-02T08:34:49.719 INFO:tasks.workunit.client.0.smithi116.stdout: "luminous", 2023-11-02T08:34:49.720 INFO:tasks.workunit.client.0.smithi116.stdout: "mimic", 2023-11-02T08:34:49.720 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap-prune", 2023-11-02T08:34:49.720 INFO:tasks.workunit.client.0.smithi116.stdout: "nautilus", 2023-11-02T08:34:49.720 INFO:tasks.workunit.client.0.smithi116.stdout: "octopus", 2023-11-02T08:34:49.720 INFO:tasks.workunit.client.0.smithi116.stdout: "pacific", 2023-11-02T08:34:49.721 INFO:tasks.workunit.client.0.smithi116.stdout: "elector-pinging", 2023-11-02T08:34:49.721 INFO:tasks.workunit.client.0.smithi116.stdout: "quincy", 2023-11-02T08:34:49.721 INFO:tasks.workunit.client.0.smithi116.stdout: "reef" 2023-11-02T08:34:49.721 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:34:49.721 INFO:tasks.workunit.client.0.smithi116.stdout: "optional": [] 2023-11-02T08:34:49.721 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.722 INFO:tasks.workunit.client.0.smithi116.stdout: "mons": [ 2023-11-02T08:34:49.722 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.722 INFO:tasks.workunit.client.0.smithi116.stdout: "rank": 0, 2023-11-02T08:34:49.722 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "a", 2023-11-02T08:34:49.722 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:34:49.723 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:34:49.723 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.723 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:34:49.723 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:7115", 2023-11-02T08:34:49.723 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 0 2023-11-02T08:34:49.724 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.724 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.724 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.724 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:7115/0", 2023-11-02T08:34:49.724 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:7115/0", 2023-11-02T08:34:49.725 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 0, 2023-11-02T08:34:49.725 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 0, 2023-11-02T08:34:49.725 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_location": "{}" 2023-11-02T08:34:49.725 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.725 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.726 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.726 INFO:tasks.workunit.client.0.smithi116.stdout: "quorum": [ 2023-11-02T08:34:49.726 INFO:tasks.workunit.client.0.smithi116.stdout: 0 2023-11-02T08:34:49.727 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:34:49.727 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap": { 2023-11-02T08:34:49.727 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 41, 2023-11-02T08:34:49.727 INFO:tasks.workunit.client.0.smithi116.stdout: "fsid": "0c9a372f-cec7-47d9-a309-8248453aabeb", 2023-11-02T08:34:49.727 INFO:tasks.workunit.client.0.smithi116.stdout: "created": "2023-11-02T08:31:36.415434+0000", 2023-11-02T08:34:49.728 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:34:13.460177+0000", 2023-11-02T08:34:49.728 INFO:tasks.workunit.client.0.smithi116.stdout: "last_up_change": "2023-11-02T08:34:12.105761+0000", 2023-11-02T08:34:49.728 INFO:tasks.workunit.client.0.smithi116.stdout: "last_in_change": "2023-11-02T08:34:13.117043+0000", 2023-11-02T08:34:49.728 INFO:tasks.workunit.client.0.smithi116.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-11-02T08:34:49.728 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_num": 5800384, 2023-11-02T08:34:49.728 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_set": [ 2023-11-02T08:34:49.729 INFO:tasks.workunit.client.0.smithi116.stdout: "nodown", 2023-11-02T08:34:49.729 INFO:tasks.workunit.client.0.smithi116.stdout: "noin", 2023-11-02T08:34:49.729 INFO:tasks.workunit.client.0.smithi116.stdout: "noout", 2023-11-02T08:34:49.729 INFO:tasks.workunit.client.0.smithi116.stdout: "pglog_hardlimit", 2023-11-02T08:34:49.729 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snapdirs", 2023-11-02T08:34:49.730 INFO:tasks.workunit.client.0.smithi116.stdout: "recovery_deletes", 2023-11-02T08:34:49.730 INFO:tasks.workunit.client.0.smithi116.stdout: "sortbitwise" 2023-11-02T08:34:49.730 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:34:49.730 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_version": 9, 2023-11-02T08:34:49.730 INFO:tasks.workunit.client.0.smithi116.stdout: "full_ratio": 0.99000000953674316, 2023-11-02T08:34:49.731 INFO:tasks.workunit.client.0.smithi116.stdout: "backfillfull_ratio": 0.99000000953674316, 2023-11-02T08:34:49.731 INFO:tasks.workunit.client.0.smithi116.stdout: "nearfull_ratio": 0.85000002384185791, 2023-11-02T08:34:49.731 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_snapshot": "", 2023-11-02T08:34:49.731 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_max": 1, 2023-11-02T08:34:49.731 INFO:tasks.workunit.client.0.smithi116.stdout: "max_osd": 3, 2023-11-02T08:34:49.731 INFO:tasks.workunit.client.0.smithi116.stdout: "require_min_compat_client": "luminous", 2023-11-02T08:34:49.732 INFO:tasks.workunit.client.0.smithi116.stdout: "min_compat_client": "jewel", 2023-11-02T08:34:49.732 INFO:tasks.workunit.client.0.smithi116.stdout: "require_osd_release": "reef", 2023-11-02T08:34:49.732 INFO:tasks.workunit.client.0.smithi116.stdout: "allow_crimson": false, 2023-11-02T08:34:49.732 INFO:tasks.workunit.client.0.smithi116.stdout: "pools": [ 2023-11-02T08:34:49.732 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.733 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": 1, 2023-11-02T08:34:49.733 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_name": "test", 2023-11-02T08:34:49.733 INFO:tasks.workunit.client.0.smithi116.stdout: "create_time": "2023-11-02T08:33:18.198986+0000", 2023-11-02T08:34:49.733 INFO:tasks.workunit.client.0.smithi116.stdout: "flags": 1, 2023-11-02T08:34:49.733 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_names": "hashpspool", 2023-11-02T08:34:49.733 INFO:tasks.workunit.client.0.smithi116.stdout: "type": 1, 2023-11-02T08:34:49.734 INFO:tasks.workunit.client.0.smithi116.stdout: "size": 3, 2023-11-02T08:34:49.734 INFO:tasks.workunit.client.0.smithi116.stdout: "min_size": 2, 2023-11-02T08:34:49.734 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_rule": 0, 2023-11-02T08:34:49.734 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_count": 0, 2023-11-02T08:34:49.734 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_target": 0, 2023-11-02T08:34:49.735 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_barrier": 0, 2023-11-02T08:34:49.735 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-11-02T08:34:49.735 INFO:tasks.workunit.client.0.smithi116.stdout: "object_hash": 2, 2023-11-02T08:34:49.735 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_autoscale_mode": "off", 2023-11-02T08:34:49.736 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num": 1, 2023-11-02T08:34:49.736 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_placement_num": 1, 2023-11-02T08:34:49.736 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_placement_num_target": 1, 2023-11-02T08:34:49.736 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num_target": 1, 2023-11-02T08:34:49.736 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num_pending": 1, 2023-11-02T08:34:49.737 INFO:tasks.workunit.client.0.smithi116.stdout: "last_pg_merge_meta": { 2023-11-02T08:34:49.737 INFO:tasks.workunit.client.0.smithi116.stdout: "source_pgid": "0.0", 2023-11-02T08:34:49.737 INFO:tasks.workunit.client.0.smithi116.stdout: "ready_epoch": 0, 2023-11-02T08:34:49.738 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_started": 0, 2023-11-02T08:34:49.738 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_clean": 0, 2023-11-02T08:34:49.738 INFO:tasks.workunit.client.0.smithi116.stdout: "source_version": "0'0", 2023-11-02T08:34:49.738 INFO:tasks.workunit.client.0.smithi116.stdout: "target_version": "0'0" 2023-11-02T08:34:49.738 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.738 INFO:tasks.workunit.client.0.smithi116.stdout: "last_change": "29", 2023-11-02T08:34:49.739 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend": "0", 2023-11-02T08:34:49.739 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend_prenautilus": "0", 2023-11-02T08:34:49.739 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend_preluminous": "0", 2023-11-02T08:34:49.739 INFO:tasks.workunit.client.0.smithi116.stdout: "auid": 0, 2023-11-02T08:34:49.740 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_mode": "selfmanaged", 2023-11-02T08:34:49.740 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_seq": 0, 2023-11-02T08:34:49.740 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_epoch": 0, 2023-11-02T08:34:49.740 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_snaps": [], 2023-11-02T08:34:49.740 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_snaps": "[]", 2023-11-02T08:34:49.741 INFO:tasks.workunit.client.0.smithi116.stdout: "quota_max_bytes": 0, 2023-11-02T08:34:49.741 INFO:tasks.workunit.client.0.smithi116.stdout: "quota_max_objects": 0, 2023-11-02T08:34:49.741 INFO:tasks.workunit.client.0.smithi116.stdout: "tiers": [], 2023-11-02T08:34:49.741 INFO:tasks.workunit.client.0.smithi116.stdout: "tier_of": -1, 2023-11-02T08:34:49.741 INFO:tasks.workunit.client.0.smithi116.stdout: "read_tier": -1, 2023-11-02T08:34:49.741 INFO:tasks.workunit.client.0.smithi116.stdout: "write_tier": -1, 2023-11-02T08:34:49.742 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_mode": "none", 2023-11-02T08:34:49.742 INFO:tasks.workunit.client.0.smithi116.stdout: "target_max_bytes": 0, 2023-11-02T08:34:49.742 INFO:tasks.workunit.client.0.smithi116.stdout: "target_max_objects": 0, 2023-11-02T08:34:49.742 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-11-02T08:34:49.742 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-11-02T08:34:49.743 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_full_ratio_micro": 800000, 2023-11-02T08:34:49.743 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_min_flush_age": 0, 2023-11-02T08:34:49.743 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_min_evict_age": 0, 2023-11-02T08:34:49.743 INFO:tasks.workunit.client.0.smithi116.stdout: "erasure_code_profile": "", 2023-11-02T08:34:49.743 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_params": { 2023-11-02T08:34:49.744 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "none" 2023-11-02T08:34:49.744 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.744 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_period": 0, 2023-11-02T08:34:49.744 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_count": 0, 2023-11-02T08:34:49.744 INFO:tasks.workunit.client.0.smithi116.stdout: "use_gmt_hitset": true, 2023-11-02T08:34:49.744 INFO:tasks.workunit.client.0.smithi116.stdout: "min_read_recency_for_promote": 0, 2023-11-02T08:34:49.745 INFO:tasks.workunit.client.0.smithi116.stdout: "min_write_recency_for_promote": 0, 2023-11-02T08:34:49.745 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_grade_decay_rate": 0, 2023-11-02T08:34:49.745 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_search_last_n": 0, 2023-11-02T08:34:49.745 INFO:tasks.workunit.client.0.smithi116.stdout: "grade_table": [], 2023-11-02T08:34:49.745 INFO:tasks.workunit.client.0.smithi116.stdout: "stripe_width": 0, 2023-11-02T08:34:49.745 INFO:tasks.workunit.client.0.smithi116.stdout: "expected_num_objects": 0, 2023-11-02T08:34:49.746 INFO:tasks.workunit.client.0.smithi116.stdout: "fast_read": false, 2023-11-02T08:34:49.746 INFO:tasks.workunit.client.0.smithi116.stdout: "options": {}, 2023-11-02T08:34:49.746 INFO:tasks.workunit.client.0.smithi116.stdout: "application_metadata": {}, 2023-11-02T08:34:49.746 INFO:tasks.workunit.client.0.smithi116.stdout: "read_balance": { 2023-11-02T08:34:49.746 INFO:tasks.workunit.client.0.smithi116.stdout: "score_acting": 2.9900000095367432, 2023-11-02T08:34:49.747 INFO:tasks.workunit.client.0.smithi116.stdout: "score_stable": 2.9900000095367432, 2023-11-02T08:34:49.747 INFO:tasks.workunit.client.0.smithi116.stdout: "optimal_score": 0.67000001668930054, 2023-11-02T08:34:49.747 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_score_acting": 2, 2023-11-02T08:34:49.748 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_score_stable": 2, 2023-11-02T08:34:49.748 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity_weighted": 1, 2023-11-02T08:34:49.748 INFO:tasks.workunit.client.0.smithi116.stdout: "average_primary_affinity": 1, 2023-11-02T08:34:49.748 INFO:tasks.workunit.client.0.smithi116.stdout: "average_primary_affinity_weighted": 1 2023-11-02T08:34:49.748 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.749 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.749 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:34:49.749 INFO:tasks.workunit.client.0.smithi116.stdout: "osds": [ 2023-11-02T08:34:49.749 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.749 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:34:49.749 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "c042b72e-f368-4127-8669-1a40dd6d2fba", 2023-11-02T08:34:49.750 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 1, 2023-11-02T08:34:49.750 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:34:49.750 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:34:49.750 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:34:49.750 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 5, 2023-11-02T08:34:49.751 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 29, 2023-11-02T08:34:49.751 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 36, 2023-11-02T08:34:49.751 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 40, 2023-11-02T08:34:49.751 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 30, 2023-11-02T08:34:49.751 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:34:49.751 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:34:49.752 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:34:49.752 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.752 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:34:49.752 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6802", 2023-11-02T08:34:49.752 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 93866 2023-11-02T08:34:49.753 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.753 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.753 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:34:49.753 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6803", 2023-11-02T08:34:49.753 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 93866 2023-11-02T08:34:49.753 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.754 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.754 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.754 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:34:49.754 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:34:49.754 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.755 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:34:49.755 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6804", 2023-11-02T08:34:49.755 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 93866 2023-11-02T08:34:49.755 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.755 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.755 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:34:49.756 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6805", 2023-11-02T08:34:49.756 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 93866 2023-11-02T08:34:49.756 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.756 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.756 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.757 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:34:49.757 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:34:49.757 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.757 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:34:49.757 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6808", 2023-11-02T08:34:49.758 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 93866 2023-11-02T08:34:49.758 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.758 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.758 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:34:49.759 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6809", 2023-11-02T08:34:49.759 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 93866 2023-11-02T08:34:49.759 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.759 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.759 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.760 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:34:49.760 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:34:49.760 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.760 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:34:49.760 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6806", 2023-11-02T08:34:49.760 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 93866 2023-11-02T08:34:49.761 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.761 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.761 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:34:49.761 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6807", 2023-11-02T08:34:49.761 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 93866 2023-11-02T08:34:49.762 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.762 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.762 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.762 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6803/93866", 2023-11-02T08:34:49.762 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6805/93866", 2023-11-02T08:34:49.762 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6809/93866", 2023-11-02T08:34:49.763 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6807/93866", 2023-11-02T08:34:49.763 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:34:49.763 INFO:tasks.workunit.client.0.smithi116.stdout: "exists", 2023-11-02T08:34:49.763 INFO:tasks.workunit.client.0.smithi116.stdout: "up" 2023-11-02T08:34:49.763 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.764 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.764 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.764 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:34:49.764 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "4ae6c215-0e13-4808-b2e7-66467c5e6ec3", 2023-11-02T08:34:49.764 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 0, 2023-11-02T08:34:49.765 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:34:49.765 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:34:49.765 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:34:49.765 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 0, 2023-11-02T08:34:49.765 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 0, 2023-11-02T08:34:49.766 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 10, 2023-11-02T08:34:49.766 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 23, 2023-11-02T08:34:49.766 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 31, 2023-11-02T08:34:49.766 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:34:49.766 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:34:49.767 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:34:49.767 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.767 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:34:49.767 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6810", 2023-11-02T08:34:49.767 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 87885 2023-11-02T08:34:49.767 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.768 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.768 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:34:49.768 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6811", 2023-11-02T08:34:49.769 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 87885 2023-11-02T08:34:49.769 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.769 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.769 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.769 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:34:49.770 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:34:49.770 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.770 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:34:49.770 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6812", 2023-11-02T08:34:49.770 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 87885 2023-11-02T08:34:49.770 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.771 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.771 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:34:49.771 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6813", 2023-11-02T08:34:49.771 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 87885 2023-11-02T08:34:49.771 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.772 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.772 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.772 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:34:49.772 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:34:49.772 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.772 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:34:49.773 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6816", 2023-11-02T08:34:49.773 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 87885 2023-11-02T08:34:49.773 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.773 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.773 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:34:49.773 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6817", 2023-11-02T08:34:49.774 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 87885 2023-11-02T08:34:49.774 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.774 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.774 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.774 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:34:49.775 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:34:49.775 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.775 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:34:49.775 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6814", 2023-11-02T08:34:49.775 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 87885 2023-11-02T08:34:49.776 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.776 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.776 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:34:49.776 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6815", 2023-11-02T08:34:49.776 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 87885 2023-11-02T08:34:49.776 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.777 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.777 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.777 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6811/87885", 2023-11-02T08:34:49.777 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6813/87885", 2023-11-02T08:34:49.777 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6817/87885", 2023-11-02T08:34:49.778 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6815/87885", 2023-11-02T08:34:49.778 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:34:49.778 INFO:tasks.workunit.client.0.smithi116.stdout: "exists" 2023-11-02T08:34:49.778 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.779 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.779 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.779 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:34:49.779 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "f289b1b9-b765-4d2c-a617-39df3b43faae", 2023-11-02T08:34:49.779 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 1, 2023-11-02T08:34:49.780 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:34:49.780 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:34:49.780 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:34:49.780 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 15, 2023-11-02T08:34:49.780 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 29, 2023-11-02T08:34:49.781 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 38, 2023-11-02T08:34:49.781 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 0, 2023-11-02T08:34:49.781 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 32, 2023-11-02T08:34:49.781 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:34:49.781 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:34:49.781 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:34:49.782 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.782 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:34:49.782 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6810", 2023-11-02T08:34:49.782 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 94410 2023-11-02T08:34:49.782 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.783 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.783 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:34:49.783 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6811", 2023-11-02T08:34:49.783 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 94410 2023-11-02T08:34:49.783 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.784 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.784 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.784 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:34:49.784 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:34:49.784 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.785 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:34:49.785 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6812", 2023-11-02T08:34:49.785 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 94410 2023-11-02T08:34:49.785 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.785 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.786 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:34:49.786 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6813", 2023-11-02T08:34:49.786 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 94410 2023-11-02T08:34:49.786 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.786 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.786 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.787 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:34:49.787 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:34:49.787 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.787 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:34:49.787 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6816", 2023-11-02T08:34:49.788 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 94410 2023-11-02T08:34:49.788 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.788 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.788 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:34:49.788 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6817", 2023-11-02T08:34:49.788 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 94410 2023-11-02T08:34:49.789 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.789 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.789 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.790 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:34:49.790 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:34:49.790 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.790 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:34:49.790 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6814", 2023-11-02T08:34:49.791 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 94410 2023-11-02T08:34:49.791 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.791 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.791 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:34:49.791 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6815", 2023-11-02T08:34:49.791 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 94410 2023-11-02T08:34:49.792 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.792 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.792 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.792 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6811/94410", 2023-11-02T08:34:49.792 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6813/94410", 2023-11-02T08:34:49.793 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6817/94410", 2023-11-02T08:34:49.793 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6815/94410", 2023-11-02T08:34:49.793 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:34:49.793 INFO:tasks.workunit.client.0.smithi116.stdout: "exists", 2023-11-02T08:34:49.793 INFO:tasks.workunit.client.0.smithi116.stdout: "up" 2023-11-02T08:34:49.793 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.794 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.794 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:34:49.794 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_xinfo": [ 2023-11-02T08:34:49.794 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.795 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:34:49.795 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:34:02.044288+0000", 2023-11-02T08:34:49.795 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:34:49.795 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:34:49.795 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:34:49.796 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:34:49.796 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:32:07.426370+0000", 2023-11-02T08:34:49.796 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:34:49.796 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.796 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.796 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:34:49.797 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:34:02.601752+0000", 2023-11-02T08:34:49.797 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:34:49.797 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:34:49.797 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:34:49.797 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:34:49.798 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:32:40.840189+0000", 2023-11-02T08:34:49.798 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:34:49.798 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.798 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.798 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:34:49.798 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:34:03.197842+0000", 2023-11-02T08:34:49.799 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:34:49.799 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:34:49.799 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:34:49.800 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:34:49.800 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:33:14.526627+0000", 2023-11-02T08:34:49.800 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:34:49.800 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.800 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:34:49.801 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap": [], 2023-11-02T08:34:49.801 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap_items": [], 2023-11-02T08:34:49.801 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap_primaries": [], 2023-11-02T08:34:49.801 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_temp": [], 2023-11-02T08:34:49.801 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_temp": [], 2023-11-02T08:34:49.802 INFO:tasks.workunit.client.0.smithi116.stdout: "blocklist": {}, 2023-11-02T08:34:49.802 INFO:tasks.workunit.client.0.smithi116.stdout: "range_blocklist": {}, 2023-11-02T08:34:49.802 INFO:tasks.workunit.client.0.smithi116.stdout: "erasure_code_profiles": { 2023-11-02T08:34:49.802 INFO:tasks.workunit.client.0.smithi116.stdout: "default": { 2023-11-02T08:34:49.802 INFO:tasks.workunit.client.0.smithi116.stdout: "k": "2", 2023-11-02T08:34:49.802 INFO:tasks.workunit.client.0.smithi116.stdout: "m": "2", 2023-11-02T08:34:49.803 INFO:tasks.workunit.client.0.smithi116.stdout: "plugin": "jerasure", 2023-11-02T08:34:49.803 INFO:tasks.workunit.client.0.smithi116.stdout: "technique": "reed_sol_van" 2023-11-02T08:34:49.803 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.803 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.803 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_snaps_queue": [], 2023-11-02T08:34:49.804 INFO:tasks.workunit.client.0.smithi116.stdout: "new_removed_snaps": [], 2023-11-02T08:34:49.804 INFO:tasks.workunit.client.0.smithi116.stdout: "new_purged_snaps": [], 2023-11-02T08:34:49.804 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_node_flags": {}, 2023-11-02T08:34:49.804 INFO:tasks.workunit.client.0.smithi116.stdout: "device_class_flags": {}, 2023-11-02T08:34:49.804 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode": { 2023-11-02T08:34:49.804 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode_enabled": false, 2023-11-02T08:34:49.805 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_bucket_count": 0, 2023-11-02T08:34:49.805 INFO:tasks.workunit.client.0.smithi116.stdout: "degraded_stretch_mode": 0, 2023-11-02T08:34:49.805 INFO:tasks.workunit.client.0.smithi116.stdout: "recovering_stretch_mode": 0, 2023-11-02T08:34:49.805 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode_bucket": 0 2023-11-02T08:34:49.805 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.805 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.806 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_metadata": [ 2023-11-02T08:34:49.806 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.806 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:34:49.806 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:34:49.806 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6804/93866,v1:127.0.0.1:6805/93866]", 2023-11-02T08:34:49.807 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:34:49.807 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:34:49.807 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:34:49.807 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:34:49.807 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:34:49.807 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:34:49.808 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:34:49.808 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:34:49.808 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:34:49.808 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:34:49.808 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2023-11-02T08:34:49.809 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:34:49.809 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:34:49.809 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:34:49.809 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:34:49.809 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:34:49.810 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:34:49.810 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:34:49.810 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:34:49.810 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:34:49.810 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:34:49.810 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:31:42.156572Z", 2023-11-02T08:34:49.811 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:34:49.811 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:34:49.811 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:34:49.811 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:34:49.811 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:34:49.812 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:34:49.812 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:34:49.812 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6802/93866,v1:127.0.0.1:6803/93866]", 2023-11-02T08:34:49.812 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:34:49.812 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/93866,v1:127.0.0.1:6809/93866]", 2023-11-02T08:34:49.812 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/93866,v1:127.0.0.1:6807/93866]", 2023-11-02T08:34:49.813 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:34:49.813 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:34:49.813 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:34:49.813 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:34:49.813 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:34:49.814 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:34:49.814 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:34:49.814 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:34:49.814 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:34:49.814 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/0", 2023-11-02T08:34:49.814 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:34:49.815 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:34:49.815 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:34:49.815 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.815 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.815 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:34:49.816 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:34:49.816 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6812/87885,v1:127.0.0.1:6813/87885]", 2023-11-02T08:34:49.816 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:34:49.816 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:34:49.816 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:34:49.816 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:34:49.817 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:34:49.817 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:34:49.817 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:34:49.817 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:34:49.817 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:34:49.818 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:34:49.818 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2023-11-02T08:34:49.818 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:34:49.818 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:34:49.818 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:34:49.819 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:34:49.819 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:34:49.819 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:34:49.819 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:34:49.819 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:34:49.819 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:34:49.820 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:34:49.820 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:32:14.444694Z", 2023-11-02T08:34:49.820 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:34:49.820 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:34:49.820 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:34:49.820 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:34:49.821 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:34:49.821 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:34:49.821 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:34:49.821 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6810/87885,v1:127.0.0.1:6811/87885]", 2023-11-02T08:34:49.821 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:34:49.822 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/87885,v1:127.0.0.1:6817/87885]", 2023-11-02T08:34:49.822 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/87885,v1:127.0.0.1:6815/87885]", 2023-11-02T08:34:49.822 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:34:49.822 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:34:49.822 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:34:49.822 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:34:49.823 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:34:49.823 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:34:49.823 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:34:49.823 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:34:49.823 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:34:49.824 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/1", 2023-11-02T08:34:49.824 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:34:49.824 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:34:49.824 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:34:49.824 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.824 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.825 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:34:49.825 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:34:49.825 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6812/94410,v1:127.0.0.1:6813/94410]", 2023-11-02T08:34:49.825 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:34:49.826 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:34:49.826 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:34:49.826 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:34:49.826 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:34:49.826 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:34:49.827 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:34:49.827 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:34:49.827 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:34:49.827 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:34:49.827 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2023-11-02T08:34:49.827 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:34:49.828 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:34:49.828 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:34:49.828 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:34:49.828 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:34:49.828 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:34:49.829 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:34:49.829 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:34:49.829 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:34:49.829 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:34:49.829 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:32:48.308517Z", 2023-11-02T08:34:49.829 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:34:49.830 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:34:49.830 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:34:49.830 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:34:49.830 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:34:49.830 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:34:49.831 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:34:49.831 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6810/94410,v1:127.0.0.1:6811/94410]", 2023-11-02T08:34:49.831 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:34:49.831 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/94410,v1:127.0.0.1:6817/94410]", 2023-11-02T08:34:49.831 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/94410,v1:127.0.0.1:6815/94410]", 2023-11-02T08:34:49.832 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:34:49.832 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:34:49.832 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:34:49.832 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:34:49.832 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:34:49.832 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:34:49.833 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:34:49.833 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:34:49.833 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:34:49.833 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/2", 2023-11-02T08:34:49.833 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:34:49.834 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:34:49.834 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:34:49.834 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.834 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:34:49.834 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_clean_epochs": { 2023-11-02T08:34:49.834 INFO:tasks.workunit.client.0.smithi116.stdout: "min_last_epoch_clean": 0, 2023-11-02T08:34:49.835 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_clean": { 2023-11-02T08:34:49.835 INFO:tasks.workunit.client.0.smithi116.stdout: "per_pool": [] 2023-11-02T08:34:49.835 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.835 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_epochs": [ 2023-11-02T08:34:49.835 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.836 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:34:49.836 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 36 2023-11-02T08:34:49.836 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.836 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.837 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:34:49.837 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 10 2023-11-02T08:34:49.837 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.837 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.837 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:34:49.837 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 39 2023-11-02T08:34:49.838 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.838 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.838 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.838 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_first_committed": 1, 2023-11-02T08:34:49.838 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_last_committed": 41, 2023-11-02T08:34:49.839 INFO:tasks.workunit.client.0.smithi116.stdout: "crushmap": { 2023-11-02T08:34:49.839 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": [ 2023-11-02T08:34:49.839 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.839 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:34:49.839 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.0", 2023-11-02T08:34:49.840 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:34:49.840 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.840 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.840 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:34:49.840 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.1", 2023-11-02T08:34:49.841 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:34:49.841 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.841 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.841 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:34:49.841 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.2", 2023-11-02T08:34:49.841 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:34:49.842 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.842 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:34:49.842 INFO:tasks.workunit.client.0.smithi116.stdout: "types": [ 2023-11-02T08:34:49.842 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.842 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 0, 2023-11-02T08:34:49.843 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd" 2023-11-02T08:34:49.843 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.843 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.843 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:34:49.843 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "host" 2023-11-02T08:34:49.843 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.844 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.844 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 2, 2023-11-02T08:34:49.844 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "chassis" 2023-11-02T08:34:49.844 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.844 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.845 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 3, 2023-11-02T08:34:49.845 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "rack" 2023-11-02T08:34:49.845 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.845 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.845 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 4, 2023-11-02T08:34:49.845 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "row" 2023-11-02T08:34:49.846 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.846 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.846 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 5, 2023-11-02T08:34:49.846 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "pdu" 2023-11-02T08:34:49.846 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.847 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.847 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 6, 2023-11-02T08:34:49.847 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "pod" 2023-11-02T08:34:49.847 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.847 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.847 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 7, 2023-11-02T08:34:49.848 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "room" 2023-11-02T08:34:49.848 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.848 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.848 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 8, 2023-11-02T08:34:49.848 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "datacenter" 2023-11-02T08:34:49.849 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.849 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.849 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 9, 2023-11-02T08:34:49.849 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "zone" 2023-11-02T08:34:49.849 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.849 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.850 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 10, 2023-11-02T08:34:49.850 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "region" 2023-11-02T08:34:49.850 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.850 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.850 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:34:49.851 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "root" 2023-11-02T08:34:49.851 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.851 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:34:49.851 INFO:tasks.workunit.client.0.smithi116.stdout: "buckets": [ 2023-11-02T08:34:49.851 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.851 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -1, 2023-11-02T08:34:49.852 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "default", 2023-11-02T08:34:49.852 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:34:49.852 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "root", 2023-11-02T08:34:49.852 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:34:49.852 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:34:49.852 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:34:49.853 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:34:49.853 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.853 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -3, 2023-11-02T08:34:49.853 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:34:49.853 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:34:49.854 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.854 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.854 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.854 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.854 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -2, 2023-11-02T08:34:49.854 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "default~hdd", 2023-11-02T08:34:49.855 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:34:49.855 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "root", 2023-11-02T08:34:49.855 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:34:49.855 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:34:49.855 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:34:49.856 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:34:49.856 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.856 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -4, 2023-11-02T08:34:49.856 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:34:49.856 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:34:49.856 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.857 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.857 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.857 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.857 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -3, 2023-11-02T08:34:49.857 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "smithi116", 2023-11-02T08:34:49.858 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:34:49.858 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "host", 2023-11-02T08:34:49.858 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:34:49.858 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:34:49.858 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:34:49.858 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:34:49.859 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.859 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:34:49.859 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:34:49.859 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:34:49.859 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.860 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.860 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:34:49.860 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:34:49.860 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 1 2023-11-02T08:34:49.860 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.860 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.861 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:34:49.861 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:34:49.861 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 2 2023-11-02T08:34:49.861 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.861 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.862 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.862 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.862 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -4, 2023-11-02T08:34:49.862 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "smithi116~hdd", 2023-11-02T08:34:49.862 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:34:49.862 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "host", 2023-11-02T08:34:49.863 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:34:49.863 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:34:49.863 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:34:49.863 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:34:49.863 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.864 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:34:49.864 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:34:49.864 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:34:49.864 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.864 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.864 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:34:49.865 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:34:49.865 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 1 2023-11-02T08:34:49.865 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.865 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.865 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:34:49.866 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:34:49.866 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 2 2023-11-02T08:34:49.866 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.866 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.866 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.866 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:34:49.867 INFO:tasks.workunit.client.0.smithi116.stdout: "rules": [ 2023-11-02T08:34:49.867 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.867 INFO:tasks.workunit.client.0.smithi116.stdout: "rule_id": 0, 2023-11-02T08:34:49.867 INFO:tasks.workunit.client.0.smithi116.stdout: "rule_name": "replicated_rule", 2023-11-02T08:34:49.867 INFO:tasks.workunit.client.0.smithi116.stdout: "type": 1, 2023-11-02T08:34:49.868 INFO:tasks.workunit.client.0.smithi116.stdout: "steps": [ 2023-11-02T08:34:49.868 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.868 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "take", 2023-11-02T08:34:49.868 INFO:tasks.workunit.client.0.smithi116.stdout: "item": -1, 2023-11-02T08:34:49.868 INFO:tasks.workunit.client.0.smithi116.stdout: "item_name": "default" 2023-11-02T08:34:49.868 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.869 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.869 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "choose_firstn", 2023-11-02T08:34:49.869 INFO:tasks.workunit.client.0.smithi116.stdout: "num": 0, 2023-11-02T08:34:49.869 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "osd" 2023-11-02T08:34:49.869 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.870 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.870 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "emit" 2023-11-02T08:34:49.870 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.870 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:34:49.870 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.870 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:34:49.871 INFO:tasks.workunit.client.0.smithi116.stdout: "tunables": { 2023-11-02T08:34:49.871 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_local_tries": 0, 2023-11-02T08:34:49.871 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_local_fallback_tries": 0, 2023-11-02T08:34:49.871 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_total_tries": 50, 2023-11-02T08:34:49.871 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_descend_once": 1, 2023-11-02T08:34:49.872 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_vary_r": 1, 2023-11-02T08:34:49.872 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_stable": 1, 2023-11-02T08:34:49.872 INFO:tasks.workunit.client.0.smithi116.stdout: "straw_calc_version": 1, 2023-11-02T08:34:49.872 INFO:tasks.workunit.client.0.smithi116.stdout: "allowed_bucket_algs": 54, 2023-11-02T08:34:49.872 INFO:tasks.workunit.client.0.smithi116.stdout: "profile": "jewel", 2023-11-02T08:34:49.872 INFO:tasks.workunit.client.0.smithi116.stdout: "optimal_tunables": 1, 2023-11-02T08:34:49.873 INFO:tasks.workunit.client.0.smithi116.stdout: "legacy_tunables": 0, 2023-11-02T08:34:49.873 INFO:tasks.workunit.client.0.smithi116.stdout: "minimum_required_version": "jewel", 2023-11-02T08:34:49.873 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables": 1, 2023-11-02T08:34:49.873 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables2": 1, 2023-11-02T08:34:49.873 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v2_rules": 0, 2023-11-02T08:34:49.873 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables3": 1, 2023-11-02T08:34:49.874 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v3_rules": 0, 2023-11-02T08:34:49.874 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v4_buckets": 1, 2023-11-02T08:34:49.874 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables5": 1, 2023-11-02T08:34:49.874 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v5_rules": 0 2023-11-02T08:34:49.874 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.875 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_args": {} 2023-11-02T08:34:49.875 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.875 INFO:tasks.workunit.client.0.smithi116.stdout: "fsmap": { 2023-11-02T08:34:49.875 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 1, 2023-11-02T08:34:49.875 INFO:tasks.workunit.client.0.smithi116.stdout: "default_fscid": -1, 2023-11-02T08:34:49.875 INFO:tasks.workunit.client.0.smithi116.stdout: "compat": { 2023-11-02T08:34:49.876 INFO:tasks.workunit.client.0.smithi116.stdout: "compat": {}, 2023-11-02T08:34:49.876 INFO:tasks.workunit.client.0.smithi116.stdout: "ro_compat": {}, 2023-11-02T08:34:49.876 INFO:tasks.workunit.client.0.smithi116.stdout: "incompat": { 2023-11-02T08:34:49.876 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_1": "base v0.20", 2023-11-02T08:34:49.876 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_2": "client writeable ranges", 2023-11-02T08:34:49.877 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_3": "default file layouts on dirs", 2023-11-02T08:34:49.877 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_4": "dir inode in separate object", 2023-11-02T08:34:49.877 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_5": "mds uses versioned encoding", 2023-11-02T08:34:49.877 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_6": "dirfrag is stored in omap", 2023-11-02T08:34:49.877 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_8": "no anchor table", 2023-11-02T08:34:49.877 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_9": "file layout v2", 2023-11-02T08:34:49.878 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_10": "snaprealm v2" 2023-11-02T08:34:49.878 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.878 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.878 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_flags": { 2023-11-02T08:34:49.879 INFO:tasks.workunit.client.0.smithi116.stderr:report 4175125153 2023-11-02T08:34:49.879 INFO:tasks.workunit.client.0.smithi116.stdout: "enable_multiple": true, 2023-11-02T08:34:49.879 INFO:tasks.workunit.client.0.smithi116.stdout: "ever_enabled_multiple": true 2023-11-02T08:34:49.880 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.880 INFO:tasks.workunit.client.0.smithi116.stdout: "standbys": [], 2023-11-02T08:34:49.880 INFO:tasks.workunit.client.0.smithi116.stdout: "filesystems": [] 2023-11-02T08:34:49.880 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.880 INFO:tasks.workunit.client.0.smithi116.stdout: "mdsmap_first_committed": 1, 2023-11-02T08:34:49.880 INFO:tasks.workunit.client.0.smithi116.stdout: "mdsmap_last_committed": 1, 2023-11-02T08:34:49.881 INFO:tasks.workunit.client.0.smithi116.stdout: "auth": { 2023-11-02T08:34:49.881 INFO:tasks.workunit.client.0.smithi116.stdout: "first_committed": 1, 2023-11-02T08:34:49.881 INFO:tasks.workunit.client.0.smithi116.stdout: "last_committed": 5, 2023-11-02T08:34:49.881 INFO:tasks.workunit.client.0.smithi116.stdout: "num_secrets": 3 2023-11-02T08:34:49.881 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.882 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg": 1, 2023-11-02T08:34:49.882 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_active": 1, 2023-11-02T08:34:49.882 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_unknown": 0, 2023-11-02T08:34:49.882 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osd": 3, 2023-11-02T08:34:49.882 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_sum": { 2023-11-02T08:34:49.882 INFO:tasks.workunit.client.0.smithi116.stdout: "stat_sum": { 2023-11-02T08:34:49.883 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes": 48300, 2023-11-02T08:34:49.883 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects": 100, 2023-11-02T08:34:49.883 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_clones": 0, 2023-11-02T08:34:49.883 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_copies": 300, 2023-11-02T08:34:49.883 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing_on_primary": 0, 2023-11-02T08:34:49.884 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing": 0, 2023-11-02T08:34:49.884 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_degraded": 100, 2023-11-02T08:34:49.884 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_misplaced": 0, 2023-11-02T08:34:49.884 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_unfound": 0, 2023-11-02T08:34:49.884 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_dirty": 100, 2023-11-02T08:34:49.884 INFO:tasks.workunit.client.0.smithi116.stdout: "num_whiteouts": 0, 2023-11-02T08:34:49.885 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read": 0, 2023-11-02T08:34:49.885 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read_kb": 0, 2023-11-02T08:34:49.885 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write": 100, 2023-11-02T08:34:49.885 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write_kb": 100, 2023-11-02T08:34:49.885 INFO:tasks.workunit.client.0.smithi116.stdout: "num_scrub_errors": 0, 2023-11-02T08:34:49.886 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shallow_scrub_errors": 0, 2023-11-02T08:34:49.886 INFO:tasks.workunit.client.0.smithi116.stdout: "num_deep_scrub_errors": 0, 2023-11-02T08:34:49.886 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_recovered": 0, 2023-11-02T08:34:49.886 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_recovered": 0, 2023-11-02T08:34:49.886 INFO:tasks.workunit.client.0.smithi116.stdout: "num_keys_recovered": 0, 2023-11-02T08:34:49.886 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_omap": 0, 2023-11-02T08:34:49.887 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_hit_set_archive": 0, 2023-11-02T08:34:49.887 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_hit_set_archive": 0, 2023-11-02T08:34:49.887 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush": 0, 2023-11-02T08:34:49.887 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_kb": 0, 2023-11-02T08:34:49.887 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict": 0, 2023-11-02T08:34:49.888 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_kb": 0, 2023-11-02T08:34:49.888 INFO:tasks.workunit.client.0.smithi116.stdout: "num_promote": 0, 2023-11-02T08:34:49.888 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_high": 0, 2023-11-02T08:34:49.888 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_low": 0, 2023-11-02T08:34:49.888 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_some": 0, 2023-11-02T08:34:49.888 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_full": 0, 2023-11-02T08:34:49.889 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_pinned": 0, 2023-11-02T08:34:49.889 INFO:tasks.workunit.client.0.smithi116.stdout: "num_legacy_snapsets": 0, 2023-11-02T08:34:49.889 INFO:tasks.workunit.client.0.smithi116.stdout: "num_large_omap_objects": 0, 2023-11-02T08:34:49.889 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_manifest": 0, 2023-11-02T08:34:49.889 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_bytes": 0, 2023-11-02T08:34:49.889 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_keys": 0, 2023-11-02T08:34:49.890 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_repaired": 0 2023-11-02T08:34:49.890 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.890 INFO:tasks.workunit.client.0.smithi116.stdout: "store_stats": { 2023-11-02T08:34:49.890 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 0, 2023-11-02T08:34:49.890 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 0, 2023-11-02T08:34:49.891 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:34:49.891 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 0, 2023-11-02T08:34:49.891 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 0, 2023-11-02T08:34:49.891 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:34:49.891 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:34:49.892 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:34:49.892 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 0, 2023-11-02T08:34:49.892 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 0 2023-11-02T08:34:49.892 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.892 INFO:tasks.workunit.client.0.smithi116.stdout: "log_size": 16, 2023-11-02T08:34:49.892 INFO:tasks.workunit.client.0.smithi116.stdout: "ondisk_log_size": 16, 2023-11-02T08:34:49.893 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 2, 2023-11-02T08:34:49.893 INFO:tasks.workunit.client.0.smithi116.stdout: "acting": 2, 2023-11-02T08:34:49.894 INFO:tasks.workunit.client.0.smithi116.stdout: "num_store_stats": 0 2023-11-02T08:34:49.894 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.894 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_sum": { 2023-11-02T08:34:49.894 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 0, 2023-11-02T08:34:49.895 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 0, 2023-11-02T08:34:49.895 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pgs": 3, 2023-11-02T08:34:49.895 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osds": 3, 2023-11-02T08:34:49.895 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_osds": 3, 2023-11-02T08:34:49.895 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_omap_osds": 3, 2023-11-02T08:34:49.895 INFO:tasks.workunit.client.0.smithi116.stdout: "kb": 314572800, 2023-11-02T08:34:49.896 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used": 83288, 2023-11-02T08:34:49.896 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_data": 2108, 2023-11-02T08:34:49.896 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_omap": 2, 2023-11-02T08:34:49.896 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_meta": 81149, 2023-11-02T08:34:49.896 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_avail": 314489512, 2023-11-02T08:34:49.897 INFO:tasks.workunit.client.0.smithi116.stdout: "statfs": { 2023-11-02T08:34:49.897 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 322122547200, 2023-11-02T08:34:49.897 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 322037260288, 2023-11-02T08:34:49.897 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:34:49.897 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 2158592, 2023-11-02T08:34:49.897 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 419117, 2023-11-02T08:34:49.898 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:34:49.898 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:34:49.898 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:34:49.898 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 2366, 2023-11-02T08:34:49.898 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 83097282 2023-11-02T08:34:49.899 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.899 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_peers": [], 2023-11-02T08:34:49.899 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_trim_queue_len": 0, 2023-11-02T08:34:49.899 INFO:tasks.workunit.client.0.smithi116.stdout: "num_snap_trimming": 0, 2023-11-02T08:34:49.899 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shards_repaired": 1, 2023-11-02T08:34:49.899 INFO:tasks.workunit.client.0.smithi116.stdout: "op_queue_age_hist": { 2023-11-02T08:34:49.900 INFO:tasks.workunit.client.0.smithi116.stdout: "histogram": [], 2023-11-02T08:34:49.900 INFO:tasks.workunit.client.0.smithi116.stdout: "upper_bound": 1 2023-11-02T08:34:49.900 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.900 INFO:tasks.workunit.client.0.smithi116.stdout: "perf_stat": { 2023-11-02T08:34:49.900 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ms": 0, 2023-11-02T08:34:49.901 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ms": 0, 2023-11-02T08:34:49.901 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ns": 0, 2023-11-02T08:34:49.901 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ns": 0 2023-11-02T08:34:49.901 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.901 INFO:tasks.workunit.client.0.smithi116.stdout: "alerts": [], 2023-11-02T08:34:49.901 INFO:tasks.workunit.client.0.smithi116.stdout: "network_ping_times": [] 2023-11-02T08:34:49.902 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.902 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_sum_by_class": { 2023-11-02T08:34:49.902 INFO:tasks.workunit.client.0.smithi116.stdout: "hdd": { 2023-11-02T08:34:49.902 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 0, 2023-11-02T08:34:49.902 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 0, 2023-11-02T08:34:49.903 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pgs": 3, 2023-11-02T08:34:49.903 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osds": 3, 2023-11-02T08:34:49.903 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_osds": 3, 2023-11-02T08:34:49.903 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_omap_osds": 3, 2023-11-02T08:34:49.903 INFO:tasks.workunit.client.0.smithi116.stdout: "kb": 314572800, 2023-11-02T08:34:49.903 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used": 83288, 2023-11-02T08:34:49.904 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_data": 2108, 2023-11-02T08:34:49.904 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_omap": 2, 2023-11-02T08:34:49.904 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_meta": 81149, 2023-11-02T08:34:49.904 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_avail": 314489512, 2023-11-02T08:34:49.904 INFO:tasks.workunit.client.0.smithi116.stdout: "statfs": { 2023-11-02T08:34:49.905 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 322122547200, 2023-11-02T08:34:49.905 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 322037260288, 2023-11-02T08:34:49.905 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:34:49.905 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 2158592, 2023-11-02T08:34:49.905 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 419117, 2023-11-02T08:34:49.906 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:34:49.906 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:34:49.906 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:34:49.906 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 2366, 2023-11-02T08:34:49.906 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 83097282 2023-11-02T08:34:49.907 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.907 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_peers": [], 2023-11-02T08:34:49.907 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_trim_queue_len": 0, 2023-11-02T08:34:49.907 INFO:tasks.workunit.client.0.smithi116.stdout: "num_snap_trimming": 0, 2023-11-02T08:34:49.907 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shards_repaired": 1, 2023-11-02T08:34:49.908 INFO:tasks.workunit.client.0.smithi116.stdout: "op_queue_age_hist": { 2023-11-02T08:34:49.908 INFO:tasks.workunit.client.0.smithi116.stdout: "histogram": [], 2023-11-02T08:34:49.908 INFO:tasks.workunit.client.0.smithi116.stdout: "upper_bound": 1 2023-11-02T08:34:49.908 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.908 INFO:tasks.workunit.client.0.smithi116.stdout: "perf_stat": { 2023-11-02T08:34:49.908 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ms": 0, 2023-11-02T08:34:49.909 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ms": 0, 2023-11-02T08:34:49.909 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ns": 0, 2023-11-02T08:34:49.909 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ns": 0 2023-11-02T08:34:49.909 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.909 INFO:tasks.workunit.client.0.smithi116.stdout: "alerts": [], 2023-11-02T08:34:49.910 INFO:tasks.workunit.client.0.smithi116.stdout: "network_ping_times": [] 2023-11-02T08:34:49.910 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.910 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.910 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_stats": [ 2023-11-02T08:34:49.910 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.910 INFO:tasks.workunit.client.0.smithi116.stdout: "poolid": 1, 2023-11-02T08:34:49.911 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg": 1, 2023-11-02T08:34:49.911 INFO:tasks.workunit.client.0.smithi116.stdout: "stat_sum": { 2023-11-02T08:34:49.911 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes": 48300, 2023-11-02T08:34:49.911 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects": 100, 2023-11-02T08:34:49.911 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_clones": 0, 2023-11-02T08:34:49.912 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_copies": 300, 2023-11-02T08:34:49.912 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing_on_primary": 0, 2023-11-02T08:34:49.912 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing": 0, 2023-11-02T08:34:49.912 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_degraded": 100, 2023-11-02T08:34:49.912 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_misplaced": 0, 2023-11-02T08:34:49.912 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_unfound": 0, 2023-11-02T08:34:49.913 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_dirty": 100, 2023-11-02T08:34:49.913 INFO:tasks.workunit.client.0.smithi116.stdout: "num_whiteouts": 0, 2023-11-02T08:34:49.913 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read": 0, 2023-11-02T08:34:49.913 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read_kb": 0, 2023-11-02T08:34:49.913 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write": 100, 2023-11-02T08:34:49.914 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write_kb": 100, 2023-11-02T08:34:49.914 INFO:tasks.workunit.client.0.smithi116.stdout: "num_scrub_errors": 0, 2023-11-02T08:34:49.914 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shallow_scrub_errors": 0, 2023-11-02T08:34:49.914 INFO:tasks.workunit.client.0.smithi116.stdout: "num_deep_scrub_errors": 0, 2023-11-02T08:34:49.914 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_recovered": 0, 2023-11-02T08:34:49.914 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_recovered": 0, 2023-11-02T08:34:49.915 INFO:tasks.workunit.client.0.smithi116.stdout: "num_keys_recovered": 0, 2023-11-02T08:34:49.915 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_omap": 0, 2023-11-02T08:34:49.915 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_hit_set_archive": 0, 2023-11-02T08:34:49.915 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_hit_set_archive": 0, 2023-11-02T08:34:49.915 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush": 0, 2023-11-02T08:34:49.916 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_kb": 0, 2023-11-02T08:34:49.916 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict": 0, 2023-11-02T08:34:49.916 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_kb": 0, 2023-11-02T08:34:49.916 INFO:tasks.workunit.client.0.smithi116.stdout: "num_promote": 0, 2023-11-02T08:34:49.916 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_high": 0, 2023-11-02T08:34:49.916 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_low": 0, 2023-11-02T08:34:49.917 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_some": 0, 2023-11-02T08:34:49.917 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_full": 0, 2023-11-02T08:34:49.917 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_pinned": 0, 2023-11-02T08:34:49.917 INFO:tasks.workunit.client.0.smithi116.stdout: "num_legacy_snapsets": 0, 2023-11-02T08:34:49.917 INFO:tasks.workunit.client.0.smithi116.stdout: "num_large_omap_objects": 0, 2023-11-02T08:34:49.918 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_manifest": 0, 2023-11-02T08:34:49.918 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_bytes": 0, 2023-11-02T08:34:49.918 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_keys": 0, 2023-11-02T08:34:49.918 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_repaired": 0 2023-11-02T08:34:49.918 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.918 INFO:tasks.workunit.client.0.smithi116.stdout: "store_stats": { 2023-11-02T08:34:49.919 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 0, 2023-11-02T08:34:49.919 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 0, 2023-11-02T08:34:49.919 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:34:49.919 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 1228800, 2023-11-02T08:34:49.919 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 144900, 2023-11-02T08:34:49.919 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:34:49.920 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:34:49.920 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:34:49.920 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 0, 2023-11-02T08:34:49.920 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 0 2023-11-02T08:34:49.920 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.921 INFO:tasks.workunit.client.0.smithi116.stdout: "log_size": 16, 2023-11-02T08:34:49.921 INFO:tasks.workunit.client.0.smithi116.stdout: "ondisk_log_size": 16, 2023-11-02T08:34:49.921 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 2, 2023-11-02T08:34:49.921 INFO:tasks.workunit.client.0.smithi116.stdout: "acting": 2, 2023-11-02T08:34:49.921 INFO:tasks.workunit.client.0.smithi116.stdout: "num_store_stats": 3 2023-11-02T08:34:49.921 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.922 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:34:49.922 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_stats": [ 2023-11-02T08:34:49.922 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.922 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:34:49.922 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 154618822667 2023-11-02T08:34:49.923 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.923 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.923 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:34:49.923 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 42949672985 2023-11-02T08:34:49.923 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.923 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.924 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:34:49.924 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 163208757258 2023-11-02T08:34:49.924 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.924 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:34:49.924 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_by_state": [ 2023-11-02T08:34:49.925 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.925 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "active+undersized+degraded", 2023-11-02T08:34:49.925 INFO:tasks.workunit.client.0.smithi116.stdout: "num": 1 2023-11-02T08:34:49.925 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.925 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:34:49.926 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_by_osd": [ 2023-11-02T08:34:49.926 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.926 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:34:49.926 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 0, 2023-11-02T08:34:49.926 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 1, 2023-11-02T08:34:49.927 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:34:49.927 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.927 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.927 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:34:49.927 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 1, 2023-11-02T08:34:49.927 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 1, 2023-11-02T08:34:49.928 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:34:49.928 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.928 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.928 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:34:49.928 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 0, 2023-11-02T08:34:49.929 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 0, 2023-11-02T08:34:49.929 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:34:49.929 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.929 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:34:49.929 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snaps": [ 2023-11-02T08:34:49.930 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:34:49.930 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": 1, 2023-11-02T08:34:49.930 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snaps": {} 2023-11-02T08:34:49.930 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.930 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:34:49.930 INFO:tasks.workunit.client.0.smithi116.stdout: "servicemap": { 2023-11-02T08:34:49.931 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 4, 2023-11-02T08:34:49.931 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:33:42.987368+0000", 2023-11-02T08:34:49.931 INFO:tasks.workunit.client.0.smithi116.stdout: "services": { 2023-11-02T08:34:49.931 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": { 2023-11-02T08:34:49.931 INFO:tasks.workunit.client.0.smithi116.stdout: "daemons": { 2023-11-02T08:34:49.931 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": "", 2023-11-02T08:34:49.932 INFO:tasks.workunit.client.0.smithi116.stdout: "2": { 2023-11-02T08:34:49.932 INFO:tasks.workunit.client.0.smithi116.stdout: "start_epoch": 0, 2023-11-02T08:34:49.932 INFO:tasks.workunit.client.0.smithi116.stdout: "start_stamp": "0.000000", 2023-11-02T08:34:49.932 INFO:tasks.workunit.client.0.smithi116.stdout: "gid": 0, 2023-11-02T08:34:49.932 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "(unrecognized address family 0)/0", 2023-11-02T08:34:49.933 INFO:tasks.workunit.client.0.smithi116.stdout: "metadata": {}, 2023-11-02T08:34:49.933 INFO:tasks.workunit.client.0.smithi116.stdout: "task_status": {} 2023-11-02T08:34:49.933 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.933 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.933 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.933 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.934 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:34:49.934 INFO:tasks.workunit.client.0.smithi116.stdout: "mgrstat_first_committed": 1, 2023-11-02T08:34:49.934 INFO:tasks.workunit.client.0.smithi116.stdout: "mgrstat_last_committed": 86, 2023-11-02T08:34:49.934 INFO:tasks.workunit.client.0.smithi116.stdout: "logm_first_committed": 1, 2023-11-02T08:34:49.934 INFO:tasks.workunit.client.0.smithi116.stdout: "logm_last_committed": 377, 2023-11-02T08:34:49.935 INFO:tasks.workunit.client.0.smithi116.stdout: "paxos": { 2023-11-02T08:34:49.935 INFO:tasks.workunit.client.0.smithi116.stdout: "first_committed": 1, 2023-11-02T08:34:49.935 INFO:tasks.workunit.client.0.smithi116.stdout: "last_committed": 543, 2023-11-02T08:34:49.935 INFO:tasks.workunit.client.0.smithi116.stdout: "last_pn": 0, 2023-11-02T08:34:49.935 INFO:tasks.workunit.client.0.smithi116.stdout: "accepted_pn": 0 2023-11-02T08:34:49.935 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:34:49.936 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T08:34:49.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2023-11-02T08:34:49.937 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2023-11-02T08:34:49.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2023-11-02T08:34:49.937 INFO:tasks.workunit.client.0.smithi116.stdout:writing non-divergent object existing_4 2023-11-02T08:34:49.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2023-11-02T08:34:49.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2023-11-02T08:34:50.104 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:34:50.104 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2023-11-02T08:34:13.711735+0000 29'100 41:122 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:33:18.225100+0000 0'0 2023-11-02T08:33:18.225100+0000 0 0 periodic scrub scheduled @ 2023-11-03T19:13:46.160155+0000 0 0 2023-11-02T08:34:50.104 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:34:50.105 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:34:50.105 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:34:50.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2023-11-02T08:34:50.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2023-11-02T08:34:50.207 INFO:tasks.workunit.client.0.smithi116.stdout:delay recovery 2023-11-02T08:34:50.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2023-11-02T08:34:50.564 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:34:50.564 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2023-11-02T08:34:13.711735+0000 29'100 41:122 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:33:18.225100+0000 0'0 2023-11-02T08:33:18.225100+0000 0 0 periodic scrub scheduled @ 2023-11-03T19:13:46.160155+0000 0 0 2023-11-02T08:34:50.564 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:34:50.565 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:34:50.565 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:34:50.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2023-11-02T08:34:50.576 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2023-11-02T08:34:50.576 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-11-02T08:34:50.576 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-11-02T08:34:50.576 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:34:50.576 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:34:50.577 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:34:50.577 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.0.asok 2023-11-02T08:34:50.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2023-11-02T08:34:50.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.0.asok set_recovery_delay 100000 2023-11-02T08:34:50.655 INFO:tasks.workunit.client.0.smithi116.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2023-11-02T08:34:50.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2023-11-02T08:34:50.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2023-11-02T08:34:50.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-11-02T08:34:50.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-11-02T08:34:50.664 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:34:50.664 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:34:50.665 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:34:50.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.2.asok 2023-11-02T08:34:50.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2023-11-02T08:34:50.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.2.asok set_recovery_delay 100000 2023-11-02T08:34:50.744 INFO:tasks.workunit.client.0.smithi116.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2023-11-02T08:34:50.751 INFO:tasks.workunit.client.0.smithi116.stdout:revive divergent 1 2023-11-02T08:34:50.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2023-11-02T08:34:50.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2023-11-02T08:34:51.106 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:34:51.106 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2023-11-02T08:34:13.711735+0000 29'100 41:122 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:33:18.225100+0000 0'0 2023-11-02T08:33:18.225100+0000 0 0 periodic scrub scheduled @ 2023-11-03T19:13:46.160155+0000 0 0 2023-11-02T08:34:51.106 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:34:51.107 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:34:51.107 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:34:51.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2023-11-02T08:34:51.800 INFO:tasks.workunit.client.0.smithi116.stderr:noup is set 2023-11-02T08:34:51.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2023-11-02T08:34:51.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2023-11-02T08:34:51.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:34:51.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2023-11-02T08:34:51.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:34:51.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2023-11-02T08:34:51.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:34:51.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:34:51.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:34:51.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:34:51.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2023-11-02T08:34:51.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2023-11-02T08:34:51.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:34:51.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:34:51.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:34:51.818 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:34:51.818 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:34:51.818 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:34:51.818 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:34:51.819 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:34:51.819 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:34:51.819 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:34:51.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:34:51.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:34:51.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:34:51.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:34:51.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:34:51.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:34:51.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:34:51.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:34:51.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:34:51.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:34:51.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2023-11-02T08:34:51.823 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T08:34:51.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2023-11-02T08:34:51.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:34:51.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2023-11-02T08:34:51.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2023-11-02T08:34:51.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:34:51.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:34:51.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:34:51.847 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:34:51.844+0000 7fcd96a63500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:34:51.848 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:34:51.846+0000 7fcd96a63500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:34:51.850 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:34:51.847+0000 7fcd96a63500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:34:52.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2023-11-02T08:34:52.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2023-11-02T08:34:52.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:34:52.361+0000 7fcd96a63500 -1 Falling back to public interface 2023-11-02T08:34:54.990 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:34:54.986+0000 7fcd96a63500 -1 osd.1 29 log_to_monitors true 2023-11-02T08:34:57.266 INFO:tasks.workunit.client.0.smithi116.stdout:delay recovery divergent 2023-11-02T08:34:57.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2023-11-02T08:34:57.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2023-11-02T08:34:57.633 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:34:57.633 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 100 0 0 47947 0 0 17 84 17 active+undersized+degraded 2023-11-02T08:34:13.711735+0000 41'101 42:125 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:33:18.225100+0000 0'0 2023-11-02T08:33:18.225100+0000 0 0 periodic scrub scheduled @ 2023-11-03T19:13:46.160155+0000 0 0 2023-11-02T08:34:57.634 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:34:57.634 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:34:57.634 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:34:57.644 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2023-11-02T08:34:57.644 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-11-02T08:34:57.645 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-11-02T08:34:57.645 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:34:57.645 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:34:57.645 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:34:57.645 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.1.asok 2023-11-02T08:34:57.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2023-11-02T08:34:57.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.1.asok set_recovery_delay 100000 2023-11-02T08:34:57.725 INFO:tasks.workunit.client.0.smithi116.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2023-11-02T08:34:57.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2023-11-02T08:34:58.329 INFO:tasks.workunit.client.0.smithi116.stderr:noup is unset 2023-11-02T08:34:58.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2023-11-02T08:34:58.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:34:58.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:34:58.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:34:58.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:34:58.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:34:58.341 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:34:58.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:34:58.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:34:58.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:34:58.909 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/93866,v1:127.0.0.1:6803/93866] [v2:127.0.0.1:6804/93866,v1:127.0.0.1:6805/93866] exists,up c042b72e-f368-4127-8669-1a40dd6d2fba 2023-11-02T08:34:58.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:34:58.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:34:58.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:34:58.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2023-11-02T08:34:58.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:34:58.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:34:58.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:34:58.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:34:58.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:34:58.912 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:34:58.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:34:58.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:34:58.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:34:59.312 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/96029,v1:127.0.0.1:6819/96029] [v2:127.0.0.1:6820/96029,v1:127.0.0.1:6821/96029] exists,up 4ae6c215-0e13-4808-b2e7-66467c5e6ec3 2023-11-02T08:34:59.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:34:59.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:34:59.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:34:59.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2023-11-02T08:34:59.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:34:59.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T08:34:59.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:34:59.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:34:59.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:34:59.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:34:59.315 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:34:59.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:34:59.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:34:59.713 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/94410,v1:127.0.0.1:6811/94410] [v2:127.0.0.1:6812/94410,v1:127.0.0.1:6813/94410] exists,up f289b1b9-b765-4d2c-a617-39df3b43faae 2023-11-02T08:34:59.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:34:59.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:34:59.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:34:59.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2023-11-02T08:35:00.077 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:35:00.077 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 100 0 0 47947 0 0 17 84 17 active+undersized+degraded 2023-11-02T08:34:13.711735+0000 41'101 43:126 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:33:18.225100+0000 0'0 2023-11-02T08:33:18.225100+0000 0 0 periodic scrub scheduled @ 2023-11-03T19:13:46.160155+0000 0 0 2023-11-02T08:35:00.077 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:35:00.077 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:35:00.078 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:35:00.088 INFO:tasks.workunit.client.0.smithi116.stdout:wait for peering 2023-11-02T08:35:00.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2023-11-02T08:35:00.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2023-11-02T08:35:00.445 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:35:00.446 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 100 0 0 47947 0 0 17 84 17 active+undersized+degraded 2023-11-02T08:34:13.711735+0000 41'101 43:126 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:33:18.225100+0000 0'0 2023-11-02T08:33:18.225100+0000 0 0 periodic scrub scheduled @ 2023-11-03T19:13:46.160155+0000 0 0 2023-11-02T08:35:00.446 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:35:00.446 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:35:00.446 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:35:00.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2023-11-02T08:35:00.562 INFO:tasks.workunit.client.0.smithi116.stdout:killing divergent 1 2023-11-02T08:35:00.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2023-11-02T08:35:00.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2023-11-02T08:35:00.916 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:35:00.916 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 100 0 0 47947 0 0 17 84 17 active+undersized+degraded 2023-11-02T08:34:13.711735+0000 41'101 43:126 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:33:18.225100+0000 0'0 2023-11-02T08:33:18.225100+0000 0 0 periodic scrub scheduled @ 2023-11-03T19:13:46.160155+0000 0 0 2023-11-02T08:35:00.917 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:35:00.917 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:35:00.917 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:35:00.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2023-11-02T08:35:00.933 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:35:00.933 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:35:00.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:35:00.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:35:00.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:35:01.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:35:01.033 INFO:tasks.workunit.client.0.smithi116.stdout:reviving divergent 1 2023-11-02T08:35:01.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2023-11-02T08:35:01.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2023-11-02T08:35:01.401 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:35:01.401 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 2 2 0 0 47947 0 0 17 84 17 active+recovering+degraded 2023-11-02T08:34:59.147595+0000 41'101 47:135 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:33:18.225100+0000 0'0 2023-11-02T08:33:18.225100+0000 0 0 periodic scrub scheduled @ 2023-11-03T11:57:42.747458+0000 0 0 2023-11-02T08:35:01.402 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:35:01.402 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:35:01.402 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:35:01.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2023-11-02T08:35:01.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2023-11-02T08:35:01.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:35:01.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2023-11-02T08:35:01.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:35:01.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2023-11-02T08:35:01.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:35:01.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:35:01.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:35:01.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:35:01.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2023-11-02T08:35:01.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2023-11-02T08:35:01.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:35:01.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:35:01.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:35:01.416 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:35:01.416 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:35:01.416 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:35:01.416 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:35:01.417 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:35:01.417 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:35:01.417 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:35:01.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:35:01.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:35:01.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:35:01.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:35:01.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:35:01.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:35:01.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:35:01.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:35:01.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:35:01.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:35:01.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2023-11-02T08:35:01.420 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T08:35:01.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2023-11-02T08:35:01.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:35:01.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2023-11-02T08:35:01.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2023-11-02T08:35:01.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:35:01.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:35:01.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:35:01.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:01.441+0000 7f3848dfb500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:35:01.447 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:01.443+0000 7f3848dfb500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:35:01.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:01.444+0000 7f3848dfb500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:35:01.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2023-11-02T08:35:01.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:35:01.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:35:01.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:35:01.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:35:01.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:35:01.836 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:35:01.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:35:01.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:35:01.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:35:01.967 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:01.963+0000 7f3848dfb500 -1 Falling back to public interface 2023-11-02T08:35:02.214 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/96029,v1:127.0.0.1:6819/96029] [v2:127.0.0.1:6820/96029,v1:127.0.0.1:6821/96029] exists,up 4ae6c215-0e13-4808-b2e7-66467c5e6ec3 2023-11-02T08:35:02.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:35:02.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:35:02.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:35:02.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2023-11-02T08:35:04.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:04.441+0000 7f3848dfb500 -1 osd.1 47 log_to_monitors true 2023-11-02T08:35:22.216 INFO:tasks.workunit.client.0.smithi116.stdout:allowing recovery 2023-11-02T08:35:22.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2023-11-02T08:35:22.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2023-11-02T08:35:22.584 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:35:22.585 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 101 0 0 0 0 48430 0 0 18 84 18 active+clean 2023-11-02T08:35:05.832465+0000 47'102 51:151 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:33:18.225100+0000 0'0 2023-11-02T08:33:18.225100+0000 0 0 periodic scrub scheduled @ 2023-11-03T19:32:37.472164+0000 0 0 2023-11-02T08:35:22.585 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:35:22.586 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:35:22.586 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:35:22.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2023-11-02T08:35:22.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2023-11-02T08:35:22.704 INFO:tasks.workunit.client.0.smithi116.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2023-11-02T08:35:22.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2023-11-02T08:35:22.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2023-11-02T08:35:22.823 INFO:tasks.workunit.client.0.smithi116.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2023-11-02T08:35:22.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2023-11-02T08:35:22.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2023-11-02T08:35:22.941 INFO:tasks.workunit.client.0.smithi116.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2023-11-02T08:35:22.950 INFO:tasks.workunit.client.0.smithi116.stdout:reading divergent objects 2023-11-02T08:35:22.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2023-11-02T08:35:22.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2023-11-02T08:35:23.312 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:35:23.312 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 101 0 0 0 0 48430 0 0 18 84 18 active+clean 2023-11-02T08:35:05.832465+0000 47'102 51:151 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:33:18.225100+0000 0'0 2023-11-02T08:33:18.225100+0000 0 0 periodic scrub scheduled @ 2023-11-03T19:32:37.472164+0000 0 0 2023-11-02T08:35:23.312 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:35:23.312 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:35:23.313 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:35:23.323 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2023-11-02T08:35:23.324 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2023-11-02T08:35:23.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2023-11-02T08:35:23.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2023-11-02T08:35:23.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2023-11-02T08:35:23.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2023-11-02T08:35:23.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2023-11-02T08:35:23.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2023-11-02T08:35:23.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2023-11-02T08:35:23.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2023-11-02T08:35:23.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2023-11-02T08:35:23.488 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2023-11-02T08:35:23.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log 2023-11-02T08:35:23.495 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:34:59.061+0000 7fcd7a9d4640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4620.0:1 2023-11-02T08:33:50.650692+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:35:23.495 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:34:59.061+0000 7fcd7a9d4640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4620.0:1 2023-11-02T08:33:50.650692+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:35:23.495 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:34:59.061+0000 7fcd7a9d4640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2023-11-02T08:35:23.495 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:34:59.061+0000 7fcd7a9d4640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'101 2023-11-02T08:35:23.495 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:34:59.061+0000 7fcd7a9d4640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2023-11-02T08:35:23.496 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:34:59.061+0000 7fcd7a9d4640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2023-11-02T08:35:23.496 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:34:59.061+0000 7fcd7a9d4640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'101 2023-11-02T08:35:23.496 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:34:59.061+0000 7fcd7a9d4640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4620.0:1 2023-11-02T08:33:50.650692+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:35:23.496 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:34:59.061+0000 7fcd7a9d4640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2023-11-02T08:35:23.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2023-11-02T08:35:23.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log 2023-11-02T08:35:23.501 INFO:tasks.workunit.client.0.smithi116.stdout:success 2023-11-02T08:35:23.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2023-11-02T08:35:23.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2023-11-02T08:35:23.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2023-11-02T08:35:23.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2023-11-02T08:35:24.112 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test' does not exist 2023-11-02T08:35:24.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2023-11-02T08:35:24.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:35:24.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:35:24.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:35:24.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:35:24.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:35:24.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:35:24.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2023-11-02T08:35:24.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2023-11-02T08:35:24.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T08:35:24.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2023-11-02T08:35:24.233 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:35:24.233 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:35:24.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:35:24.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:35:24.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:35:24.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:35:24.236 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T08:35:24.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T08:35:24.237 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T08:35:24.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T08:35:24.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T08:35:24.239 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T08:35:24.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:35:24.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T08:35:24.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T08:35:24.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:35:24.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T08:35:24.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T08:35:24.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2023-11-02T08:35:24.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T08:35:24.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:35:24.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:35:24.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.85082 2023-11-02T08:35:24.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T08:35:24.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T08:35:24.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2023-11-02T08:35:24.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2023-11-02T08:35:24.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2023-11-02T08:35:24.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2023-11-02T08:35:24.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2023-11-02T08:35:24.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T08:35:24.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2023-11-02T08:35:24.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:35:24.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:35:24.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:35:24.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:35:24.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:35:24.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:35:24.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T08:35:24.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T08:35:24.272 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T08:35:24.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T08:35:24.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T08:35:24.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T08:35:24.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:35:24.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T08:35:24.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T08:35:24.274 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:35:24.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T08:35:24.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T08:35:24.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2023-11-02T08:35:24.274 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T08:35:24.275 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:35:24.275 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:35:24.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.85082 2023-11-02T08:35:24.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T08:35:24.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T08:35:24.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2023-11-02T08:35:24.276 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T08:35:24.276 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:35:24.276 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:35:24.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.85082 2023-11-02T08:35:24.277 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T08:35:24.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-11-02T08:35:24.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T08:35:24.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2023-11-02T08:35:24.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2023-11-02T08:35:24.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2023-11-02T08:35:24.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2023-11-02T08:35:24.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2023-11-02T08:35:24.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2023-11-02T08:35:24.279 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2023-11-02T08:35:24.280 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2023-11-02T08:35:24.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2023-11-02T08:35:24.281 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:35:24.281 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:35:24.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2023-11-02T08:35:24.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2023-11-02T08:35:24.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T08:35:24.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T08:35:24.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T08:35:24.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2023-11-02T08:35:24.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2023-11-02T08:35:24.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T08:35:24.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:35:24.652 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:35:24.653 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:35:24.653 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:35:24.653 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:35:24.653 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:35:24.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T08:35:24.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T08:35:24.837 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T08:35:24.837 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T08:35:24.837 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T08:35:24.838 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T08:35:24.838 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T08:35:24.838 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T08:35:24.839 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T08:35:24.839 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T08:35:24.840 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:35:24.840 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:35:24.840 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:35:24.841 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-mon.a.asok 2023-11-02T08:35:24.841 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T08:35:24.841 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.85082/ceph-mon.a.asok config get fsid 2023-11-02T08:35:24.914 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T08:35:24.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T08:35:24.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T08:35:24.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T08:35:24.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T08:35:24.915 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T08:35:24.916 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T08:35:24.916 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T08:35:24.916 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:35:24.916 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:35:24.916 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:35:24.917 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-mon.a.asok 2023-11-02T08:35:24.917 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T08:35:24.917 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.85082/ceph-mon.a.asok config get mon_host 2023-11-02T08:35:24.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2023-11-02T08:35:24.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2023-11-02T08:35:24.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T08:35:24.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T08:35:24.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T08:35:24.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2023-11-02T08:35:24.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T08:35:25.847 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T08:35:25.848 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:35:25.848 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:35:25.849 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:35:25.849 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:35:25.849 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:35:25.850 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:35:25.850 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T08:35:25.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T08:35:25.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2023-11-02T08:35:25.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2023-11-02T08:35:25.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2023-11-02T08:35:25.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:35:25.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T08:35:25.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:35:25.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2023-11-02T08:35:25.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:35:25.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:35:25.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:35:25.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:35:25.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2023-11-02T08:35:25.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2023-11-02T08:35:25.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:35:25.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:35:25.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:35:25.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:35:25.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:35:25.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:35:25.888 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:35:25.888 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:35:25.888 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:35:25.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:35:25.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:35:25.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:35:25.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:35:25.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:35:25.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:35:25.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:35:25.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:35:25.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:35:25.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:35:25.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:35:25.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:35:25.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:35:25.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2023-11-02T08:35:25.893 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:35:25.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b4aaa3b4-59f6-4f52-81b9-50100a210eb6 2023-11-02T08:35:25.895 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 b4aaa3b4-59f6-4f52-81b9-50100a210eb6 2023-11-02T08:35:25.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b4aaa3b4-59f6-4f52-81b9-50100a210eb6' 2023-11-02T08:35:25.896 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:35:25.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBNX0Nlo2a0NRAAYlhFTFMcSmotF9uG9DdBCA== 2023-11-02T08:35:25.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBNX0Nlo2a0NRAAYlhFTFMcSmotF9uG9DdBCA=="}' 2023-11-02T08:35:25.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b4aaa3b4-59f6-4f52-81b9-50100a210eb6 -i td/divergent-priors/0/new.json 2023-11-02T08:35:26.261 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:35:26.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2023-11-02T08:35:26.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNX0Nlo2a0NRAAYlhFTFMcSmotF9uG9DdBCA== --osd-uuid b4aaa3b4-59f6-4f52-81b9-50100a210eb6 2023-11-02T08:35:26.306 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:26.303+0000 7f147100e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:35:26.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:26.305+0000 7f147100e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:35:26.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:26.306+0000 7f147100e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:35:26.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:26.306+0000 7f147100e500 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2023-11-02T08:35:26.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:26.306+0000 7f147100e500 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2023-11-02T08:35:26.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:26.306+0000 7f147100e500 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2023-11-02T08:35:26.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:26.381+0000 7f147100e500 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2023-11-02T08:35:31.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2023-11-02T08:35:31.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:35:31.694 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T08:35:31.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T08:35:31.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:35:32.219 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T08:35:32.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T08:35:32.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:35:32.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:35:32.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:35:32.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:35:32.252 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:32.249+0000 7f432a0d2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:35:32.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:32.259+0000 7f432a0d2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:35:32.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:32.260+0000 7f432a0d2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:35:32.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T08:35:32.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:35:32.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:35:32.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:35:32.615 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:35:32.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:35:32.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:35:32.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:35:32.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:35:32.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:35:32.797 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:32.794+0000 7f432a0d2500 -1 Falling back to public interface 2023-11-02T08:35:33.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:35:34.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:35:34.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:35:34.028 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:35:34.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:35:34.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:35:34.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:35:34.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:35:35.072 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:35.069+0000 7f432a0d2500 -1 osd.0 0 log_to_monitors true 2023-11-02T08:35:35.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:35:35.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:35:35.415 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:35:35.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:35:35.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:35:35.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:35:35.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:35:36.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:35:36.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:35:36.978 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:35:36.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:35:36.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:35:36.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:35:37.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:35:38.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:35:38.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:35:38.382 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:35:38.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:35:38.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:35:38.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:35:38.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:35:39.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:35:39.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:35:39.888 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:35:39.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:35:39.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:35:39.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:35:40.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:35:41.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:35:41.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:35:41.281 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:35:41.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:35:41.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:35:41.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:35:41.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:35:42.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:35:42.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:35:42.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:35:42.735 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:35:42.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:35:42.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:35:43.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:35:44.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:35:44.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:35:44.119 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:35:44.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:35:44.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:35:44.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:35:44.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:35:45.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:35:45.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:35:45.519 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:35:45.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:35:45.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:35:45.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:35:46.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:35:47.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:35:47.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:35:47.024 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:35:47.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:35:47.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:35:47.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:35:47.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:35:48.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:35:48.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:35:48.668 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:35:48.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:35:48.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:35:48.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:35:49.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:35:50.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:35:50.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:35:50.194 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:35:50.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:35:50.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:35:50.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:35:50.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:35:51.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:35:51.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:35:51.889 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:35:51.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:35:51.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:35:51.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:35:52.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:35:53.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:35:53.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:35:53.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:35:53.319 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:35:53.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:35:53.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:35:54.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:35:55.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:35:55.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:35:55.117 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:35:55.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:35:55.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:35:55.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:35:55.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:35:56.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:35:56.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:35:56.725 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:35:56.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:35:56.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:35:56.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:35:57.110 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/98580,v1:127.0.0.1:6803/98580] [v2:127.0.0.1:6804/98580,v1:127.0.0.1:6805/98580] exists,up b4aaa3b4-59f6-4f52-81b9-50100a210eb6 2023-11-02T08:35:57.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:35:57.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:35:57.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:35:57.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2023-11-02T08:35:57.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2023-11-02T08:35:57.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2023-11-02T08:35:57.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:35:57.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-11-02T08:35:57.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:35:57.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2023-11-02T08:35:57.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:35:57.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:35:57.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:35:57.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:35:57.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2023-11-02T08:35:57.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2023-11-02T08:35:57.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:35:57.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:35:57.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:35:57.119 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:35:57.119 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:35:57.119 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:35:57.120 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:35:57.120 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:35:57.120 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:35:57.120 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:35:57.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:35:57.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:35:57.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:35:57.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:35:57.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:35:57.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:35:57.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:35:57.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:35:57.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:35:57.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:35:57.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:35:57.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:35:57.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2023-11-02T08:35:57.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:35:57.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3aac1df4-eb52-4ae6-947d-2e4756e32297 2023-11-02T08:35:57.124 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 3aac1df4-eb52-4ae6-947d-2e4756e32297 2023-11-02T08:35:57.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3aac1df4-eb52-4ae6-947d-2e4756e32297' 2023-11-02T08:35:57.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:35:57.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBtX0NlOyf4BxAAUTis2Od17kzM609HmjuFnw== 2023-11-02T08:35:57.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBtX0NlOyf4BxAAUTis2Od17kzM609HmjuFnw=="}' 2023-11-02T08:35:57.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3aac1df4-eb52-4ae6-947d-2e4756e32297 -i td/divergent-priors/1/new.json 2023-11-02T08:35:57.623 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:35:57.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2023-11-02T08:35:57.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBtX0NlOyf4BxAAUTis2Od17kzM609HmjuFnw== --osd-uuid 3aac1df4-eb52-4ae6-947d-2e4756e32297 2023-11-02T08:35:57.669 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:57.666+0000 7f046720c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:35:57.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:57.668+0000 7f046720c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:35:57.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:57.669+0000 7f046720c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:35:57.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:57.669+0000 7f046720c500 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2023-11-02T08:35:57.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:57.669+0000 7f046720c500 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2023-11-02T08:35:57.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:57.669+0000 7f046720c500 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2023-11-02T08:35:57.747 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:35:57.744+0000 7f046720c500 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2023-11-02T08:36:03.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2023-11-02T08:36:03.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:36:03.454 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2023-11-02T08:36:03.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-11-02T08:36:03.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:36:03.978 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T08:36:03.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-11-02T08:36:03.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:36:03.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:36:03.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:36:03.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:36:04.012 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:36:04.007+0000 7f174f40d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:36:04.013 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:36:04.009+0000 7f174f40d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:36:04.014 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:36:04.010+0000 7f174f40d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:36:04.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-11-02T08:36:04.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:36:04.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:36:04.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:36:04.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:36:04.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:04.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:36:04.410 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:36:04.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:04.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:36:04.533 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:36:04.529+0000 7f174f40d500 -1 Falling back to public interface 2023-11-02T08:36:04.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:05.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:05.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:05.827 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:36:05.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:36:05.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:05.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:36:06.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:06.793 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:36:06.788+0000 7f174f40d500 -1 osd.1 0 log_to_monitors true 2023-11-02T08:36:07.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:07.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:07.221 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:36:07.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:36:07.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:07.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:36:07.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:08.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:08.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:08.733 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:36:08.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:36:08.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:08.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:36:09.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:10.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:10.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:10.135 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:36:10.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:36:10.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:10.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:36:10.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:11.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:11.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:11.643 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:36:11.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:36:11.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:11.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:36:12.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:13.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:13.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:13.028 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:36:13.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:36:13.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:13.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:36:13.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:14.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:14.501 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:36:14.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:14.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:36:14.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:14.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:36:14.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:15.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:15.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:15.909 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:36:15.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:36:15.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:15.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:36:16.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:17.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:17.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:17.453 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:36:17.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:36:17.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:17.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:36:17.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:18.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:18.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:18.852 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:36:18.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:36:18.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:18.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:36:19.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:20.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:20.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:20.248 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:36:20.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:36:20.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:20.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:36:20.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:21.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:21.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:21.855 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:36:21.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:36:21.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:21.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:36:22.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:23.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:23.655 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:36:23.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:23.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:36:23.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:23.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:36:24.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:25.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:25.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:25.378 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:36:25.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:36:25.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:25.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:36:26.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:27.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:27.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:27.091 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:36:27.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:36:27.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:27.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:36:27.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:28.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:28.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:28.820 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:36:28.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:36:28.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:28.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:36:29.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:30.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:30.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:30.379 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T08:36:30.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T08:36:30.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:30.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:36:30.760 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/100309,v1:127.0.0.1:6811/100309] [v2:127.0.0.1:6812/100309,v1:127.0.0.1:6813/100309] exists,up 3aac1df4-eb52-4ae6-947d-2e4756e32297 2023-11-02T08:36:30.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:36:30.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:36:30.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:36:30.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2023-11-02T08:36:30.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2023-11-02T08:36:30.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2023-11-02T08:36:30.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:36:30.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-11-02T08:36:30.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:36:30.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2023-11-02T08:36:30.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:36:30.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:36:30.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:36:30.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:36:30.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2023-11-02T08:36:30.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2023-11-02T08:36:30.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:36:30.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:36:30.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:36:30.776 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:36:30.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:36:30.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:36:30.777 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:36:30.778 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:36:30.778 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:36:30.779 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:36:30.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:36:30.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:36:30.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:36:30.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:36:30.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:36:30.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:36:30.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:36:30.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:36:30.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:36:30.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:36:30.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:36:30.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:36:30.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2023-11-02T08:36:30.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:36:30.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d98f6933-2821-4359-90f4-f9b40e2a8e71 2023-11-02T08:36:30.783 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 d98f6933-2821-4359-90f4-f9b40e2a8e71 2023-11-02T08:36:30.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d98f6933-2821-4359-90f4-f9b40e2a8e71' 2023-11-02T08:36:30.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:36:30.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCOX0NlsfOzLhAA9wmr3iVgXnbCkif2dYnjbQ== 2023-11-02T08:36:30.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCOX0NlsfOzLhAA9wmr3iVgXnbCkif2dYnjbQ=="}' 2023-11-02T08:36:30.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d98f6933-2821-4359-90f4-f9b40e2a8e71 -i td/divergent-priors/2/new.json 2023-11-02T08:36:31.264 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:36:31.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2023-11-02T08:36:31.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCOX0NlsfOzLhAA9wmr3iVgXnbCkif2dYnjbQ== --osd-uuid d98f6933-2821-4359-90f4-f9b40e2a8e71 2023-11-02T08:36:31.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:36:31.307+0000 7f1ddc80c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:36:31.334 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:36:31.331+0000 7f1ddc80c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:36:31.334 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:36:31.332+0000 7f1ddc80c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:36:31.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:36:31.332+0000 7f1ddc80c500 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2023-11-02T08:36:31.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:36:31.332+0000 7f1ddc80c500 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2023-11-02T08:36:31.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:36:31.332+0000 7f1ddc80c500 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2023-11-02T08:36:31.402 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:36:31.399+0000 7f1ddc80c500 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2023-11-02T08:36:37.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2023-11-02T08:36:37.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:36:37.241 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2023-11-02T08:36:37.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-11-02T08:36:37.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:36:37.764 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T08:36:37.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-11-02T08:36:37.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:36:37.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:36:37.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:36:37.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:36:37.798 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:36:37.795+0000 7f6f1cc0e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:36:37.822 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:36:37.818+0000 7f6f1cc0e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:36:37.823 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:36:37.820+0000 7f6f1cc0e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:36:38.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-11-02T08:36:38.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:36:38.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T08:36:38.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:36:38.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:36:38.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:38.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:36:38.197 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:36:38.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:38.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:36:38.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:36:38.359+0000 7f6f1cc0e500 -1 Falling back to public interface 2023-11-02T08:36:38.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:39.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:39.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:39.622 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:36:39.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:36:39.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:39.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:36:40.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:40.682 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:36:40.679+0000 7f6f1cc0e500 -1 osd.2 0 log_to_monitors true 2023-11-02T08:36:41.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:41.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:41.031 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:36:41.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:36:41.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:41.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:36:41.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:42.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:42.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:42.562 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:36:42.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:36:42.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:42.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:36:42.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:43.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:43.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:43.898 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:36:43.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:36:43.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:43.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:36:44.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:45.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:45.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:45.309 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:36:45.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:36:45.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:45.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:36:45.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:46.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:46.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:46.701 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:36:46.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:36:46.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:46.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:36:47.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:48.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:48.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:48.091 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:36:48.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:36:48.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:48.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:36:48.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:49.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:49.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:49.613 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:36:49.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:36:49.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:49.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:36:50.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:51.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:51.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:51.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:36:51.006 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:36:51.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:51.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:36:51.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:52.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:52.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:52.459 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:36:52.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:36:52.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:52.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:36:53.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:54.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:54.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:54.348 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:36:54.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:36:54.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:54.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:36:55.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:56.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:56.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:56.196 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:36:56.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:36:56.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:56.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:36:57.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:58.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:58.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:58.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:36:58.180 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:36:58.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:58.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:36:58.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:36:59.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:36:59.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:36:59.960 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:36:59.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:36:59.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:36:59.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:37:00.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:37:01.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:37:01.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:37:01.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:37:01.838 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:37:01.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:37:01.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:37:02.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:37:03.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:37:03.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:37:03.642 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:37:03.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:37:03.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:37:03.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:37:04.069 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/102048,v1:127.0.0.1:6819/102048] [v2:127.0.0.1:6820/102048,v1:127.0.0.1:6821/102048] exists,up d98f6933-2821-4359-90f4-f9b40e2a8e71 2023-11-02T08:37:04.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:37:04.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:37:04.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:37:04.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2023-11-02T08:37:04.733 INFO:tasks.workunit.client.0.smithi116.stderr:noout is set 2023-11-02T08:37:04.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2023-11-02T08:37:05.503 INFO:tasks.workunit.client.0.smithi116.stderr:noin is set 2023-11-02T08:37:05.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2023-11-02T08:37:06.341 INFO:tasks.workunit.client.0.smithi116.stderr:nodown is set 2023-11-02T08:37:06.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2023-11-02T08:37:06.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2023-11-02T08:37:07.254 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test' already exists 2023-11-02T08:37:07.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T08:37:08.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2023-11-02T08:37:09.164 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 size to 3 2023-11-02T08:37:09.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2023-11-02T08:37:09.975 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 min_size to 2 2023-11-02T08:37:09.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2023-11-02T08:37:09.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:37:09.987 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:37:10.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:37:10.361 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:37:10.361 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:37:10.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:37:10.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:37:10.362 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:37:10.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836501 2023-11-02T08:37:10.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836501 2023-11-02T08:37:10.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836501' 2023-11-02T08:37:10.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:37:10.480 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:37:10.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672975 2023-11-02T08:37:10.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672975 2023-11-02T08:37:10.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836501 1-42949672975' 2023-11-02T08:37:10.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:37:10.599 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:37:10.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509449 2023-11-02T08:37:10.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509449 2023-11-02T08:37:10.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836501 1-42949672975 2-64424509449' 2023-11-02T08:37:10.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:37:10.717 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836501 2023-11-02T08:37:10.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:37:10.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:37:10.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836501 2023-11-02T08:37:10.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:37:10.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836501 2023-11-02T08:37:10.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2023-11-02T08:37:10.720 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836501 2023-11-02T08:37:10.721 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:37:11.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836499 -lt 21474836501 2023-11-02T08:37:11.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T08:37:12.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T08:37:12.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:37:12.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836501 -lt 21474836501 2023-11-02T08:37:12.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:37:12.551 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672975 2023-11-02T08:37:12.552 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:37:12.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T08:37:12.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672975 2023-11-02T08:37:12.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:37:12.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672975 2023-11-02T08:37:12.554 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672975 2023-11-02T08:37:12.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2023-11-02T08:37:12.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T08:37:12.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672975 -lt 42949672975 2023-11-02T08:37:12.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:37:12.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509449 2023-11-02T08:37:12.970 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:37:12.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:37:12.970 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509449 2023-11-02T08:37:12.971 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:37:12.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509449 2023-11-02T08:37:12.972 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509449 2023-11-02T08:37:12.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2023-11-02T08:37:12.972 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:37:13.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509449 -lt 64424509449 2023-11-02T08:37:13.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2023-11-02T08:37:13.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T08:37:13.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T08:37:13.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T08:37:13.353 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-11-02T08:37:13.353 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T08:37:13.353 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T08:37:13.353 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T08:37:13.354 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T08:37:13.354 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T08:37:13.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-11-02T08:37:13.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T08:37:13.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T08:37:13.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T08:37:13.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:37:13.435 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:37:13.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:37:13.835 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:37:13.835 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:37:13.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:37:13.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:37:13.836 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:37:13.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836504 2023-11-02T08:37:13.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836504 2023-11-02T08:37:13.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836504' 2023-11-02T08:37:13.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:37:13.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:37:14.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672977 2023-11-02T08:37:14.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672977 2023-11-02T08:37:14.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836504 1-42949672977' 2023-11-02T08:37:14.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:37:14.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:37:14.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509452 2023-11-02T08:37:14.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509452 2023-11-02T08:37:14.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836504 1-42949672977 2-64424509452' 2023-11-02T08:37:14.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:37:14.193 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836504 2023-11-02T08:37:14.194 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:37:14.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:37:14.195 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836504 2023-11-02T08:37:14.195 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:37:14.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836504 2023-11-02T08:37:14.196 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836504 2023-11-02T08:37:14.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2023-11-02T08:37:14.196 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:37:14.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836502 -lt 21474836504 2023-11-02T08:37:14.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T08:37:15.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T08:37:15.599 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:37:15.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836504 -lt 21474836504 2023-11-02T08:37:15.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:37:15.984 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672977 2023-11-02T08:37:15.984 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:37:15.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T08:37:15.985 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672977 2023-11-02T08:37:15.985 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:37:15.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672977 2023-11-02T08:37:15.986 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672977 2023-11-02T08:37:15.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2023-11-02T08:37:15.987 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T08:37:16.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672978 -lt 42949672977 2023-11-02T08:37:16.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:37:16.394 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509452 2023-11-02T08:37:16.394 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:37:16.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:37:16.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509452 2023-11-02T08:37:16.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:37:16.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509452 2023-11-02T08:37:16.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2023-11-02T08:37:16.396 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509452 2023-11-02T08:37:16.396 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:37:16.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509452 -lt 64424509452 2023-11-02T08:37:16.802 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T08:37:16.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:37:16.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:37:17.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2023-11-02T08:37:17.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:37:17.318 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:37:17.318 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:37:17.318 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:37:17.318 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:37:17.319 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:37:17.319 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:37:17.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T08:37:17.720 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:37:17.721 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:37:17.721 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:37:18.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2023-11-02T08:37:18.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T08:37:18.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T08:37:18.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2023-11-02T08:37:18.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2023-11-02T08:37:18.591 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:37:18.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2023-11-02T08:37:18.602 INFO:tasks.workunit.client.0.smithi116.stdout:primary and soon to be divergent is 1 2023-11-02T08:37:18.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2023-11-02T08:37:18.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2023-11-02T08:37:18.958 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:37:18.959 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-02T08:37:08.164007+0000 0'0 28:13 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:37:06.739239+0000 0'0 2023-11-02T08:37:06.739239+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:22:12.222863+0000 0 0 2023-11-02T08:37:18.959 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:37:18.959 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:37:18.959 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:37:18.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2023-11-02T08:37:18.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2023-11-02T08:37:18.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2023-11-02T08:37:18.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2023-11-02T08:37:18.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2023-11-02T08:37:18.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2023-11-02T08:37:18.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2023-11-02T08:37:18.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2023-11-02T08:37:18.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2023-11-02T08:37:18.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2023-11-02T08:37:18.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2023-11-02T08:37:18.972 INFO:tasks.workunit.client.0.smithi116.stdout:writing initial objects 2023-11-02T08:37:18.974 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2023-11-02T08:37:18.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:18.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2023-11-02T08:37:19.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:19.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2023-11-02T08:37:19.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:19.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2023-11-02T08:37:19.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:19.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2023-11-02T08:37:19.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:19.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2023-11-02T08:37:19.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:19.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2023-11-02T08:37:19.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:19.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2023-11-02T08:37:20.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:20.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2023-11-02T08:37:20.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:20.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2023-11-02T08:37:20.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:20.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2023-11-02T08:37:20.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:20.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2023-11-02T08:37:20.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:20.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2023-11-02T08:37:20.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:20.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2023-11-02T08:37:20.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:20.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2023-11-02T08:37:21.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:21.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2023-11-02T08:37:21.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:21.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2023-11-02T08:37:21.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:21.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2023-11-02T08:37:21.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:21.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2023-11-02T08:37:21.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:21.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2023-11-02T08:37:21.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:21.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2023-11-02T08:37:21.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:21.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2023-11-02T08:37:22.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:22.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2023-11-02T08:37:22.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:22.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2023-11-02T08:37:22.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:22.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2023-11-02T08:37:22.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:22.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2023-11-02T08:37:22.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:22.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2023-11-02T08:37:22.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:22.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2023-11-02T08:37:22.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:22.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2023-11-02T08:37:23.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:23.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2023-11-02T08:37:23.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:23.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2023-11-02T08:37:23.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:23.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2023-11-02T08:37:23.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:23.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2023-11-02T08:37:23.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:23.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2023-11-02T08:37:23.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:23.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2023-11-02T08:37:23.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:23.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2023-11-02T08:37:24.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:24.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2023-11-02T08:37:24.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:24.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2023-11-02T08:37:24.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:24.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2023-11-02T08:37:24.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:24.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2023-11-02T08:37:24.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:24.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2023-11-02T08:37:24.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:24.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2023-11-02T08:37:24.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:24.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2023-11-02T08:37:25.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:25.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2023-11-02T08:37:25.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:25.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2023-11-02T08:37:25.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:25.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2023-11-02T08:37:25.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:25.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2023-11-02T08:37:25.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:25.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2023-11-02T08:37:25.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:25.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2023-11-02T08:37:25.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:25.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2023-11-02T08:37:26.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:26.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2023-11-02T08:37:26.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:26.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2023-11-02T08:37:26.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:26.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2023-11-02T08:37:26.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:26.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2023-11-02T08:37:26.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:26.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2023-11-02T08:37:26.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:26.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2023-11-02T08:37:27.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:27.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2023-11-02T08:37:27.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:27.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2023-11-02T08:37:27.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:27.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2023-11-02T08:37:27.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:27.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2023-11-02T08:37:27.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:27.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2023-11-02T08:37:27.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:27.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2023-11-02T08:37:27.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:27.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2023-11-02T08:37:28.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:28.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2023-11-02T08:37:28.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:28.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2023-11-02T08:37:28.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:28.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2023-11-02T08:37:28.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:28.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2023-11-02T08:37:28.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:28.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2023-11-02T08:37:28.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:28.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2023-11-02T08:37:28.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:28.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2023-11-02T08:37:29.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:29.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2023-11-02T08:37:29.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:29.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2023-11-02T08:37:29.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:29.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2023-11-02T08:37:29.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:29.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2023-11-02T08:37:29.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:29.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2023-11-02T08:37:29.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:29.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2023-11-02T08:37:29.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:29.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2023-11-02T08:37:30.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:30.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2023-11-02T08:37:30.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:30.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2023-11-02T08:37:30.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:30.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2023-11-02T08:37:30.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:30.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2023-11-02T08:37:30.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:30.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2023-11-02T08:37:30.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:30.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2023-11-02T08:37:31.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:31.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2023-11-02T08:37:31.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:31.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2023-11-02T08:37:31.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:31.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2023-11-02T08:37:31.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:31.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2023-11-02T08:37:31.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:31.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2023-11-02T08:37:31.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:31.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2023-11-02T08:37:31.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:31.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2023-11-02T08:37:32.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:32.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2023-11-02T08:37:32.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:32.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2023-11-02T08:37:32.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:32.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2023-11-02T08:37:32.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:32.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2023-11-02T08:37:32.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:32.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2023-11-02T08:37:32.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:32.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2023-11-02T08:37:33.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:33.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2023-11-02T08:37:33.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:33.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2023-11-02T08:37:33.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:33.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2023-11-02T08:37:33.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:33.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2023-11-02T08:37:33.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2023-11-02T08:37:33.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2023-11-02T08:37:33.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2023-11-02T08:37:33.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2023-11-02T08:37:33.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T08:37:33.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T08:37:33.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T08:37:33.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2023-11-02T08:37:33.741 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T08:37:33.741 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T08:37:33.741 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T08:37:33.741 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T08:37:33.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T08:37:33.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2023-11-02T08:37:33.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T08:37:33.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T08:37:33.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T08:37:33.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:37:33.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:37:34.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:37:34.200 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:37:34.200 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:37:34.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:37:34.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:37:34.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:37:34.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836510 2023-11-02T08:37:34.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836510 2023-11-02T08:37:34.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836510' 2023-11-02T08:37:34.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:37:34.320 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:37:34.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672983 2023-11-02T08:37:34.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672983 2023-11-02T08:37:34.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836510 1-42949672983' 2023-11-02T08:37:34.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:37:34.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:37:34.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509458 2023-11-02T08:37:34.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509458 2023-11-02T08:37:34.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836510 1-42949672983 2-64424509458' 2023-11-02T08:37:34.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:37:34.558 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836510 2023-11-02T08:37:34.558 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:37:34.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:37:34.559 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836510 2023-11-02T08:37:34.559 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:37:34.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836510 2023-11-02T08:37:34.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2023-11-02T08:37:34.560 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836510 2023-11-02T08:37:34.561 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:37:34.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836508 -lt 21474836510 2023-11-02T08:37:34.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T08:37:35.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T08:37:35.978 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:37:36.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836510 -lt 21474836510 2023-11-02T08:37:36.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:37:36.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672983 2023-11-02T08:37:36.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:37:36.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T08:37:36.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672983 2023-11-02T08:37:36.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:37:36.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672983 2023-11-02T08:37:36.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2023-11-02T08:37:36.480 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672983 2023-11-02T08:37:36.481 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T08:37:36.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672984 -lt 42949672983 2023-11-02T08:37:36.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:37:36.859 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509458 2023-11-02T08:37:36.859 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:37:36.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:37:36.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509458 2023-11-02T08:37:36.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:37:36.860 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509458 2023-11-02T08:37:36.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509458 2023-11-02T08:37:36.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2023-11-02T08:37:36.861 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:37:37.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509458 -lt 64424509458 2023-11-02T08:37:37.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T08:37:37.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:37:37.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:37:37.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2023-11-02T08:37:37.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:37:37.783 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:37:37.783 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:37:37.783 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:37:37.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:37:37.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:37:37.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:37:38.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T08:37:38.196 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:37:38.197 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:37:38.197 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:37:38.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2023-11-02T08:37:38.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T08:37:38.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T08:37:38.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2023-11-02T08:37:38.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2023-11-02T08:37:38.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2023-11-02T08:37:38.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2023-11-02T08:37:38.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2023-11-02T08:37:39.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2023-11-02T08:37:39.110 INFO:tasks.workunit.client.0.smithi116.stdout:blackholing osds 0 2 2023-11-02T08:37:39.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2023-11-02T08:37:39.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2023-11-02T08:37:39.479 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:37:39.479 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 0 0 0 48300 0 0 16 84 16 active+clean 2023-11-02T08:37:08.164007+0000 29'100 29:113 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:37:06.739239+0000 0'0 2023-11-02T08:37:06.739239+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:22:12.222863+0000 0 0 2023-11-02T08:37:39.479 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:37:39.479 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:37:39.480 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:37:39.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2023-11-02T08:37:39.490 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2023-11-02T08:37:39.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-11-02T08:37:39.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-11-02T08:37:39.493 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:37:39.493 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:37:39.493 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:37:39.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.0.asok 2023-11-02T08:37:39.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2023-11-02T08:37:39.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.0.asok config set objectstore_blackhole 1 2023-11-02T08:37:39.566 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T08:37:39.566 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2023-11-02T08:37:39.567 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T08:37:39.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2023-11-02T08:37:39.574 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2023-11-02T08:37:39.574 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-11-02T08:37:39.574 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-11-02T08:37:39.575 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:37:39.575 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:37:39.575 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:37:39.575 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.2.asok 2023-11-02T08:37:39.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2023-11-02T08:37:39.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.2.asok config set objectstore_blackhole 1 2023-11-02T08:37:39.650 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T08:37:39.650 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2023-11-02T08:37:39.650 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T08:37:39.656 INFO:tasks.workunit.client.0.smithi116.stdout:create new divergent objects 2023-11-02T08:37:39.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2023-11-02T08:37:39.657 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2023-11-02T08:37:39.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2023-11-02T08:37:39.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2023-11-02T08:37:39.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2023-11-02T08:37:39.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2023-11-02T08:37:39.660 INFO:tasks.workunit.client.0.smithi116.stdout:writing divergent objects 2023-11-02T08:37:39.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2023-11-02T08:37:39.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2023-11-02T08:37:39.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2023-11-02T08:37:39.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2023-11-02T08:37:39.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2023-11-02T08:37:39.663 INFO:tasks.workunit.client.0.smithi116.stdout:remove divergent objects 2023-11-02T08:37:39.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2023-11-02T08:37:39.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2023-11-02T08:37:39.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2023-11-02T08:37:39.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2023-11-02T08:37:39.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2023-11-02T08:37:39.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2023-11-02T08:37:39.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2023-11-02T08:37:39.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2023-11-02T08:37:39.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2023-11-02T08:37:39.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2023-11-02T08:37:39.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2023-11-02T08:37:39.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2023-11-02T08:37:49.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2023-11-02T08:37:49.671 INFO:tasks.workunit.client.0.smithi116.stderr:rados: no process found 2023-11-02T08:37:49.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2023-11-02T08:37:49.672 INFO:tasks.workunit.client.0.smithi116.stdout:killing all the osds 2023-11-02T08:37:49.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2023-11-02T08:37:50.036 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:37:50.036 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 0 0 0 46628 0 0 16 90 16 active+clean 2023-11-02T08:37:08.164007+0000 29'106 29:119 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:37:06.739239+0000 0'0 2023-11-02T08:37:06.739239+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:22:12.222863+0000 0 0 2023-11-02T08:37:50.037 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:37:50.037 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:37:50.037 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:37:50.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2023-11-02T08:37:50.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:37:50.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:37:50.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:37:50.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:37:50.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:37:50.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:37:50.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2023-11-02T08:37:50.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2023-11-02T08:37:50.666 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 is already down. 2023-11-02T08:37:50.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2023-11-02T08:37:50.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2023-11-02T08:37:51.236 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 is already down. 2023-11-02T08:37:51.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2023-11-02T08:37:51.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2023-11-02T08:37:51.755 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 is already down. 2023-11-02T08:37:51.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2023-11-02T08:37:51.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2023-11-02T08:37:52.405 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 is already out. 2023-11-02T08:37:52.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2023-11-02T08:37:52.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2023-11-02T08:37:52.923 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 is already out. 2023-11-02T08:37:52.934 INFO:tasks.workunit.client.0.smithi116.stdout:bringing up non_divergent 0 2 2023-11-02T08:37:52.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2023-11-02T08:37:52.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2023-11-02T08:37:53.334 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:37:53.335 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 0 0 0 46628 0 0 16 90 16 stale+active+clean 2023-11-02T08:37:08.164007+0000 29'106 29:119 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:37:06.739239+0000 0'0 2023-11-02T08:37:06.739239+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:22:12.222863+0000 0 0 2023-11-02T08:37:53.335 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:37:53.335 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:37:53.336 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:37:53.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2023-11-02T08:37:53.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2023-11-02T08:37:53.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2023-11-02T08:37:53.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:37:53.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-11-02T08:37:53.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:37:53.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2023-11-02T08:37:53.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:37:53.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:37:53.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:37:53.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:37:53.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2023-11-02T08:37:53.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2023-11-02T08:37:53.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:37:53.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:37:53.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:37:53.355 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:37:53.355 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:37:53.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:37:53.356 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:37:53.356 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:37:53.356 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:37:53.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:37:53.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:37:53.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:37:53.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:37:53.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:37:53.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:37:53.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:37:53.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:37:53.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:37:53.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:37:53.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:37:53.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2023-11-02T08:37:53.360 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T08:37:53.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-11-02T08:37:53.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:37:53.361 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2023-11-02T08:37:53.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-11-02T08:37:53.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:37:53.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:37:53.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:37:53.381 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:37:53.378+0000 7f59d9413500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:37:53.383 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:37:53.380+0000 7f59d9413500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:37:53.384 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:37:53.381+0000 7f59d9413500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:37:53.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-11-02T08:37:53.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:37:53.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:37:53.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:37:53.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:37:53.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:37:53.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:37:53.769 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:37:53.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:37:53.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:37:53.898 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:37:53.895+0000 7f59d9413500 -1 Falling back to public interface 2023-11-02T08:37:54.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:37:55.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:37:55.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:37:55.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:37:55.166 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:37:55.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:37:55.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:37:55.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:37:56.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:37:56.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:37:56.553 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:37:56.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:37:56.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:37:56.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:37:56.610 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:37:56.607+0000 7f59d9413500 -1 osd.0 29 log_to_monitors true 2023-11-02T08:37:56.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:37:57.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:37:57.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:37:57.945 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:37:57.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:37:57.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:37:57.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:37:58.333 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/106338,v1:127.0.0.1:6803/106338] [v2:127.0.0.1:6804/106338,v1:127.0.0.1:6805/106338] exists,up b4aaa3b4-59f6-4f52-81b9-50100a210eb6 2023-11-02T08:37:58.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:37:58.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:37:58.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:37:58.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2023-11-02T08:37:58.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2023-11-02T08:37:58.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2023-11-02T08:37:58.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:37:58.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2023-11-02T08:37:58.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:37:58.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2023-11-02T08:37:58.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:37:58.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:37:58.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:37:58.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:37:58.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2023-11-02T08:37:58.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2023-11-02T08:37:58.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:37:58.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:37:58.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:37:58.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:37:58.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:37:58.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:37:58.341 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:37:58.342 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:37:58.342 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:37:58.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:37:58.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:37:58.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:37:58.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:37:58.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:37:58.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:37:58.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:37:58.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:37:58.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:37:58.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:37:58.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:37:58.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2023-11-02T08:37:58.345 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T08:37:58.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2023-11-02T08:37:58.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:37:58.346 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2023-11-02T08:37:58.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2023-11-02T08:37:58.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:37:58.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:37:58.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:37:58.369 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:37:58.366+0000 7fb63aefa500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:37:58.407 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:37:58.404+0000 7fb63aefa500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:37:58.408 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:37:58.405+0000 7fb63aefa500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:37:58.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2023-11-02T08:37:58.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:37:58.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T08:37:58.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:37:58.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:37:58.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:37:58.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:37:58.722 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:37:58.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:37:58.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:37:58.930 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:37:58.927+0000 7fb63aefa500 -1 Falling back to public interface 2023-11-02T08:37:59.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:38:00.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:38:00.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:38:00.155 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:38:00.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:38:00.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:38:00.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:38:00.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:38:01.556 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:38:01.553+0000 7fb63aefa500 -1 osd.2 29 log_to_monitors true 2023-11-02T08:38:01.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:38:01.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:38:01.616 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:38:01.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:38:01.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:38:01.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:38:02.048 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/106908,v1:127.0.0.1:6811/106908] [v2:127.0.0.1:6812/106908,v1:127.0.0.1:6813/106908] exists,up d98f6933-2821-4359-90f4-f9b40e2a8e71 2023-11-02T08:38:02.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:38:02.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:38:02.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:38:02.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2023-11-02T08:38:02.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2023-11-02T08:38:02.585 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 is already in. 2023-11-02T08:38:02.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2023-11-02T08:38:02.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2023-11-02T08:38:03.314 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 is already in. 2023-11-02T08:38:03.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2023-11-02T08:38:03.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2023-11-02T08:38:03.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T08:38:03.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T08:38:03.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T08:38:03.327 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2023-11-02T08:38:03.328 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T08:38:03.329 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T08:38:03.329 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T08:38:03.329 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T08:38:03.329 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T08:38:03.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2023-11-02T08:38:03.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T08:38:03.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T08:38:03.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T08:38:03.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:38:03.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:38:03.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:38:03.758 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:38:03.759 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:38:03.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:38:03.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:38:03.760 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:38:03.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=154618822660 2023-11-02T08:38:03.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 154618822660 2023-11-02T08:38:03.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-154618822660' 2023-11-02T08:38:03.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:38:03.877 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:38:03.977 INFO:tasks.workunit.client.0.smithi116.stderr:Error ENXIO: problem getting command descriptions from osd.1 2023-11-02T08:38:03.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2023-11-02T08:38:03.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2023-11-02T08:38:03.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2023-11-02T08:38:03.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:38:03.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:38:04.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=163208757251 2023-11-02T08:38:04.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 163208757251 2023-11-02T08:38:04.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2023-11-02T08:38:04.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:38:04.101 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-154618822660 2023-11-02T08:38:04.101 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:38:04.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:38:04.102 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-154618822660 2023-11-02T08:38:04.102 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:38:04.103 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 154618822660 2023-11-02T08:38:04.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=154618822660 2023-11-02T08:38:04.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2023-11-02T08:38:04.106 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:38:04.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 154618822658 -lt 154618822660 2023-11-02T08:38:04.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T08:38:05.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T08:38:05.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:38:05.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 154618822660 -lt 154618822660 2023-11-02T08:38:05.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:38:05.965 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-163208757251 2023-11-02T08:38:05.965 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:38:05.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:38:05.966 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-163208757251 2023-11-02T08:38:05.966 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:38:05.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=163208757251 2023-11-02T08:38:05.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2023-11-02T08:38:05.967 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 163208757251 2023-11-02T08:38:05.968 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:38:06.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 163208757251 -lt 163208757251 2023-11-02T08:38:06.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T08:38:06.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:38:06.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:38:06.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2023-11-02T08:38:06.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:38:06.850 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:38:06.850 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:38:06.850 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:38:06.850 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:38:06.850 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:38:06.851 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:38:07.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:38:07.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:38:07.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:38:07.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:38:07.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:38:07.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2023-11-02T08:38:07.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2023-11-02T08:38:07.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2023-11-02T08:38:07.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:38:07.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T08:38:07.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:38:07.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:38:07.928 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:38:07.928 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:38:07.928 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:38:07.928 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:38:07.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:38:07.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:38:08.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:38:08.294 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:38:08.294 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:38:08.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:38:08.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:38:08.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:38:08.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:38:08.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:38:08.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:38:08.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:38:08.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:38:08.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:38:08.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:38:09.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:38:09.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:38:09.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2023-11-02T08:38:09.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:38:09.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2023-11-02T08:38:09.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:38:09.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:38:09.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:38:09.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:38:09.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:38:09.540 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:38:09.540 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:38:09.540 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:38:09.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:38:09.908 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:38:09.908 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:38:09.908 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:38:10.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:38:10.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:38:10.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:38:10.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:38:10.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:38:10.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:38:10.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:38:10.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:38:10.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:38:11.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:38:11.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:38:11.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2023-11-02T08:38:11.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:38:11.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2023-11-02T08:38:11.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:38:11.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:38:11.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:38:11.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:38:11.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:38:11.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:38:11.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:38:11.413 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:38:11.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:38:11.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:38:11.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:38:11.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:38:12.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:38:12.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:38:12.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:38:12.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:38:12.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:38:12.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:38:12.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:38:12.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:38:12.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:38:12.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:38:12.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:38:12.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2023-11-02T08:38:12.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:38:12.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2023-11-02T08:38:13.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:38:13.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:38:13.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:38:13.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:38:13.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:38:13.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:38:13.632 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:38:13.632 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:38:14.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:38:14.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:38:14.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:38:14.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:38:14.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:38:14.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:38:14.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:38:14.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:38:14.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:38:14.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:38:14.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:38:14.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:38:14.517 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:38:15.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:38:15.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:38:15.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2023-11-02T08:38:15.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:38:15.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2023-11-02T08:38:16.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:38:16.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:38:16.654 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:38:16.654 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:38:16.655 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:38:16.655 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:38:16.655 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:38:16.655 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:38:17.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:38:17.037 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:38:17.037 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:38:17.038 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:38:17.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:38:17.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:38:17.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:38:17.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:38:17.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:38:17.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:38:17.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:38:17.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:38:17.543 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:38:18.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:38:18.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:38:18.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2023-11-02T08:38:18.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:38:18.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2023-11-02T08:38:21.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:38:21.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:38:21.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:38:21.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:38:21.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:38:21.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:38:21.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:38:21.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:38:21.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:38:21.642 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:38:21.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:38:21.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:38:22.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:38:22.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:38:22.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:38:22.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:38:22.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:38:22.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:38:22.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:38:22.149 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:38:22.150 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:38:22.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:38:22.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:38:22.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2023-11-02T08:38:22.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:38:22.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2023-11-02T08:38:29.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:38:29.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:38:29.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:38:29.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:38:29.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:38:29.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:38:29.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:38:29.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:38:29.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:38:29.431 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:38:29.431 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:38:29.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:38:29.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:38:29.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:38:29.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:38:29.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:38:29.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:38:29.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:38:29.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:38:29.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:38:29.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:38:30.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:38:30.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:38:30.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2023-11-02T08:38:30.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:38:30.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2023-11-02T08:38:37.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:38:37.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:38:37.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:38:37.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:38:37.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:38:37.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:38:37.785 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:38:37.785 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:38:38.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:38:38.163 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:38:38.163 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:38:38.163 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:38:38.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:38:38.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:38:38.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:38:38.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:38:38.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:38:38.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:38:38.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:38:38.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:38:38.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:38:39.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:38:39.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:38:39.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2023-11-02T08:38:39.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2023-11-02T08:38:39.717 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T08:38:39.717 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_fingerprint": "d678552a-81a2-4f9f-a139-3b7833870c2a", 2023-11-02T08:38:39.717 INFO:tasks.workunit.client.0.smithi116.stdout: "version": "18.2.0-1159-gb5a9486b", 2023-11-02T08:38:39.718 INFO:tasks.workunit.client.0.smithi116.stdout: "commit": "b5a9486b4a6f144f83c51b7ec33111a2c86dc081", 2023-11-02T08:38:39.718 INFO:tasks.workunit.client.0.smithi116.stdout: "timestamp": "2023-11-02T08:38:39.713859+0000", 2023-11-02T08:38:39.718 INFO:tasks.workunit.client.0.smithi116.stdout: "tag": "", 2023-11-02T08:38:39.718 INFO:tasks.workunit.client.0.smithi116.stdout: "health": { 2023-11-02T08:38:39.718 INFO:tasks.workunit.client.0.smithi116.stdout: "status": "HEALTH_WARN", 2023-11-02T08:38:39.719 INFO:tasks.workunit.client.0.smithi116.stdout: "checks": { 2023-11-02T08:38:39.719 INFO:tasks.workunit.client.0.smithi116.stdout: "OSDMAP_FLAGS": { 2023-11-02T08:38:39.719 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:38:39.719 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:38:39.719 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "nodown,noout,noin flag(s) set", 2023-11-02T08:38:39.720 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 17 2023-11-02T08:38:39.720 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.720 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [], 2023-11-02T08:38:39.720 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:38:39.720 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.720 INFO:tasks.workunit.client.0.smithi116.stdout: "OSD_DOWN": { 2023-11-02T08:38:39.721 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:38:39.721 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:38:39.721 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "1 osds down", 2023-11-02T08:38:39.721 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 1 2023-11-02T08:38:39.721 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.722 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:38:39.722 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.722 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "osd.1 (root=default,host=smithi116) is down" 2023-11-02T08:38:39.722 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.722 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:38:39.723 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:38:39.723 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.723 INFO:tasks.workunit.client.0.smithi116.stdout: "PG_DEGRADED": { 2023-11-02T08:38:39.723 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:38:39.723 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:38:39.724 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2023-11-02T08:38:39.724 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 1 2023-11-02T08:38:39.724 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.724 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:38:39.724 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.725 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2023-11-02T08:38:39.725 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.725 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:38:39.725 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:38:39.725 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.726 INFO:tasks.workunit.client.0.smithi116.stdout: "POOL_APP_NOT_ENABLED": { 2023-11-02T08:38:39.726 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:38:39.726 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:38:39.726 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "1 pool(s) do not have an application enabled", 2023-11-02T08:38:39.726 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 1 2023-11-02T08:38:39.727 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.727 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:38:39.727 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.727 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "application not enabled on pool 'test'" 2023-11-02T08:38:39.727 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.727 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.728 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2023-11-02T08:38:39.728 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.728 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:38:39.728 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:38:39.728 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.729 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.729 INFO:tasks.workunit.client.0.smithi116.stdout: "mutes": [] 2023-11-02T08:38:39.729 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.729 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap_first_committed": 1, 2023-11-02T08:38:39.729 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap_last_committed": 1, 2023-11-02T08:38:39.729 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap": { 2023-11-02T08:38:39.730 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 1, 2023-11-02T08:38:39.730 INFO:tasks.workunit.client.0.smithi116.stdout: "fsid": "0c9a372f-cec7-47d9-a309-8248453aabeb", 2023-11-02T08:38:39.730 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:35:24.297501Z", 2023-11-02T08:38:39.730 INFO:tasks.workunit.client.0.smithi116.stdout: "created": "2023-11-02T08:35:24.297501Z", 2023-11-02T08:38:39.730 INFO:tasks.workunit.client.0.smithi116.stdout: "min_mon_release": 18, 2023-11-02T08:38:39.731 INFO:tasks.workunit.client.0.smithi116.stdout: "min_mon_release_name": "reef", 2023-11-02T08:38:39.731 INFO:tasks.workunit.client.0.smithi116.stdout: "election_strategy": 1, 2023-11-02T08:38:39.731 INFO:tasks.workunit.client.0.smithi116.stdout: "disallowed_leaders: ": "", 2023-11-02T08:38:39.731 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode": false, 2023-11-02T08:38:39.731 INFO:tasks.workunit.client.0.smithi116.stdout: "tiebreaker_mon": "", 2023-11-02T08:38:39.732 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_ranks: ": "", 2023-11-02T08:38:39.732 INFO:tasks.workunit.client.0.smithi116.stdout: "features": { 2023-11-02T08:38:39.732 INFO:tasks.workunit.client.0.smithi116.stdout: "persistent": [ 2023-11-02T08:38:39.732 INFO:tasks.workunit.client.0.smithi116.stdout: "kraken", 2023-11-02T08:38:39.732 INFO:tasks.workunit.client.0.smithi116.stdout: "luminous", 2023-11-02T08:38:39.733 INFO:tasks.workunit.client.0.smithi116.stdout: "mimic", 2023-11-02T08:38:39.733 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap-prune", 2023-11-02T08:38:39.733 INFO:tasks.workunit.client.0.smithi116.stdout: "nautilus", 2023-11-02T08:38:39.733 INFO:tasks.workunit.client.0.smithi116.stdout: "octopus", 2023-11-02T08:38:39.735 INFO:tasks.workunit.client.0.smithi116.stdout: "pacific", 2023-11-02T08:38:39.735 INFO:tasks.workunit.client.0.smithi116.stdout: "elector-pinging", 2023-11-02T08:38:39.735 INFO:tasks.workunit.client.0.smithi116.stdout: "quincy", 2023-11-02T08:38:39.735 INFO:tasks.workunit.client.0.smithi116.stdout: "reef" 2023-11-02T08:38:39.735 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:38:39.736 INFO:tasks.workunit.client.0.smithi116.stdout: "optional": [] 2023-11-02T08:38:39.736 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.736 INFO:tasks.workunit.client.0.smithi116.stdout: "mons": [ 2023-11-02T08:38:39.736 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.736 INFO:tasks.workunit.client.0.smithi116.stdout: "rank": 0, 2023-11-02T08:38:39.737 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "a", 2023-11-02T08:38:39.737 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:38:39.737 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:38:39.737 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.737 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:38:39.738 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:7115", 2023-11-02T08:38:39.738 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 0 2023-11-02T08:38:39.738 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.738 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.738 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.739 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:7115/0", 2023-11-02T08:38:39.739 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:7115/0", 2023-11-02T08:38:39.739 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 0, 2023-11-02T08:38:39.740 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 0, 2023-11-02T08:38:39.740 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_location": "{}" 2023-11-02T08:38:39.740 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.740 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.740 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.741 INFO:tasks.workunit.client.0.smithi116.stdout: "quorum": [ 2023-11-02T08:38:39.741 INFO:tasks.workunit.client.0.smithi116.stdout: 0 2023-11-02T08:38:39.741 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:38:39.741 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap": { 2023-11-02T08:38:39.741 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 42, 2023-11-02T08:38:39.742 INFO:tasks.workunit.client.0.smithi116.stdout: "fsid": "0c9a372f-cec7-47d9-a309-8248453aabeb", 2023-11-02T08:38:39.742 INFO:tasks.workunit.client.0.smithi116.stdout: "created": "2023-11-02T08:35:25.095191+0000", 2023-11-02T08:38:39.742 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:38:03.312654+0000", 2023-11-02T08:38:39.742 INFO:tasks.workunit.client.0.smithi116.stdout: "last_up_change": "2023-11-02T08:38:01.787977+0000", 2023-11-02T08:38:39.742 INFO:tasks.workunit.client.0.smithi116.stdout: "last_in_change": "2023-11-02T08:38:03.008439+0000", 2023-11-02T08:38:39.743 INFO:tasks.workunit.client.0.smithi116.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-11-02T08:38:39.743 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_num": 5800384, 2023-11-02T08:38:39.743 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_set": [ 2023-11-02T08:38:39.743 INFO:tasks.workunit.client.0.smithi116.stdout: "nodown", 2023-11-02T08:38:39.743 INFO:tasks.workunit.client.0.smithi116.stdout: "noin", 2023-11-02T08:38:39.744 INFO:tasks.workunit.client.0.smithi116.stdout: "noout", 2023-11-02T08:38:39.744 INFO:tasks.workunit.client.0.smithi116.stdout: "pglog_hardlimit", 2023-11-02T08:38:39.744 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snapdirs", 2023-11-02T08:38:39.744 INFO:tasks.workunit.client.0.smithi116.stdout: "recovery_deletes", 2023-11-02T08:38:39.744 INFO:tasks.workunit.client.0.smithi116.stdout: "sortbitwise" 2023-11-02T08:38:39.745 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:38:39.745 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_version": 9, 2023-11-02T08:38:39.745 INFO:tasks.workunit.client.0.smithi116.stdout: "full_ratio": 0.99000000953674316, 2023-11-02T08:38:39.745 INFO:tasks.workunit.client.0.smithi116.stdout: "backfillfull_ratio": 0.99000000953674316, 2023-11-02T08:38:39.745 INFO:tasks.workunit.client.0.smithi116.stdout: "nearfull_ratio": 0.85000002384185791, 2023-11-02T08:38:39.745 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_snapshot": "", 2023-11-02T08:38:39.746 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_max": 1, 2023-11-02T08:38:39.746 INFO:tasks.workunit.client.0.smithi116.stdout: "max_osd": 3, 2023-11-02T08:38:39.746 INFO:tasks.workunit.client.0.smithi116.stdout: "require_min_compat_client": "luminous", 2023-11-02T08:38:39.746 INFO:tasks.workunit.client.0.smithi116.stdout: "min_compat_client": "jewel", 2023-11-02T08:38:39.747 INFO:tasks.workunit.client.0.smithi116.stdout: "require_osd_release": "reef", 2023-11-02T08:38:39.747 INFO:tasks.workunit.client.0.smithi116.stdout: "allow_crimson": false, 2023-11-02T08:38:39.747 INFO:tasks.workunit.client.0.smithi116.stdout: "pools": [ 2023-11-02T08:38:39.747 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.747 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": 1, 2023-11-02T08:38:39.748 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_name": "test", 2023-11-02T08:38:39.748 INFO:tasks.workunit.client.0.smithi116.stdout: "create_time": "2023-11-02T08:37:06.652604+0000", 2023-11-02T08:38:39.748 INFO:tasks.workunit.client.0.smithi116.stdout: "flags": 1, 2023-11-02T08:38:39.748 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_names": "hashpspool", 2023-11-02T08:38:39.748 INFO:tasks.workunit.client.0.smithi116.stdout: "type": 1, 2023-11-02T08:38:39.749 INFO:tasks.workunit.client.0.smithi116.stdout: "size": 3, 2023-11-02T08:38:39.749 INFO:tasks.workunit.client.0.smithi116.stdout: "min_size": 2, 2023-11-02T08:38:39.749 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_rule": 0, 2023-11-02T08:38:39.749 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_count": 0, 2023-11-02T08:38:39.749 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_target": 0, 2023-11-02T08:38:39.750 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_barrier": 0, 2023-11-02T08:38:39.750 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-11-02T08:38:39.750 INFO:tasks.workunit.client.0.smithi116.stdout: "object_hash": 2, 2023-11-02T08:38:39.750 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_autoscale_mode": "off", 2023-11-02T08:38:39.750 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num": 1, 2023-11-02T08:38:39.751 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_placement_num": 1, 2023-11-02T08:38:39.751 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_placement_num_target": 1, 2023-11-02T08:38:39.751 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num_target": 1, 2023-11-02T08:38:39.751 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num_pending": 1, 2023-11-02T08:38:39.751 INFO:tasks.workunit.client.0.smithi116.stdout: "last_pg_merge_meta": { 2023-11-02T08:38:39.752 INFO:tasks.workunit.client.0.smithi116.stdout: "source_pgid": "0.0", 2023-11-02T08:38:39.752 INFO:tasks.workunit.client.0.smithi116.stdout: "ready_epoch": 0, 2023-11-02T08:38:39.752 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_started": 0, 2023-11-02T08:38:39.752 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_clean": 0, 2023-11-02T08:38:39.752 INFO:tasks.workunit.client.0.smithi116.stdout: "source_version": "0'0", 2023-11-02T08:38:39.753 INFO:tasks.workunit.client.0.smithi116.stdout: "target_version": "0'0" 2023-11-02T08:38:39.753 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.753 INFO:tasks.workunit.client.0.smithi116.stdout: "last_change": "29", 2023-11-02T08:38:39.753 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend": "0", 2023-11-02T08:38:39.753 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend_prenautilus": "0", 2023-11-02T08:38:39.754 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend_preluminous": "0", 2023-11-02T08:38:39.754 INFO:tasks.workunit.client.0.smithi116.stdout: "auid": 0, 2023-11-02T08:38:39.754 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_mode": "selfmanaged", 2023-11-02T08:38:39.754 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_seq": 0, 2023-11-02T08:38:39.754 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_epoch": 0, 2023-11-02T08:38:39.755 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_snaps": [], 2023-11-02T08:38:39.755 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_snaps": "[]", 2023-11-02T08:38:39.755 INFO:tasks.workunit.client.0.smithi116.stdout: "quota_max_bytes": 0, 2023-11-02T08:38:39.755 INFO:tasks.workunit.client.0.smithi116.stdout: "quota_max_objects": 0, 2023-11-02T08:38:39.755 INFO:tasks.workunit.client.0.smithi116.stdout: "tiers": [], 2023-11-02T08:38:39.756 INFO:tasks.workunit.client.0.smithi116.stdout: "tier_of": -1, 2023-11-02T08:38:39.756 INFO:tasks.workunit.client.0.smithi116.stdout: "read_tier": -1, 2023-11-02T08:38:39.756 INFO:tasks.workunit.client.0.smithi116.stdout: "write_tier": -1, 2023-11-02T08:38:39.756 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_mode": "none", 2023-11-02T08:38:39.756 INFO:tasks.workunit.client.0.smithi116.stdout: "target_max_bytes": 0, 2023-11-02T08:38:39.757 INFO:tasks.workunit.client.0.smithi116.stdout: "target_max_objects": 0, 2023-11-02T08:38:39.757 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-11-02T08:38:39.757 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-11-02T08:38:39.757 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_full_ratio_micro": 800000, 2023-11-02T08:38:39.757 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_min_flush_age": 0, 2023-11-02T08:38:39.758 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_min_evict_age": 0, 2023-11-02T08:38:39.758 INFO:tasks.workunit.client.0.smithi116.stdout: "erasure_code_profile": "", 2023-11-02T08:38:39.758 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_params": { 2023-11-02T08:38:39.758 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "none" 2023-11-02T08:38:39.758 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.759 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_period": 0, 2023-11-02T08:38:39.759 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_count": 0, 2023-11-02T08:38:39.759 INFO:tasks.workunit.client.0.smithi116.stdout: "use_gmt_hitset": true, 2023-11-02T08:38:39.759 INFO:tasks.workunit.client.0.smithi116.stdout: "min_read_recency_for_promote": 0, 2023-11-02T08:38:39.759 INFO:tasks.workunit.client.0.smithi116.stdout: "min_write_recency_for_promote": 0, 2023-11-02T08:38:39.760 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_grade_decay_rate": 0, 2023-11-02T08:38:39.760 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_search_last_n": 0, 2023-11-02T08:38:39.760 INFO:tasks.workunit.client.0.smithi116.stdout: "grade_table": [], 2023-11-02T08:38:39.760 INFO:tasks.workunit.client.0.smithi116.stdout: "stripe_width": 0, 2023-11-02T08:38:39.760 INFO:tasks.workunit.client.0.smithi116.stdout: "expected_num_objects": 0, 2023-11-02T08:38:39.760 INFO:tasks.workunit.client.0.smithi116.stdout: "fast_read": false, 2023-11-02T08:38:39.761 INFO:tasks.workunit.client.0.smithi116.stdout: "options": {}, 2023-11-02T08:38:39.761 INFO:tasks.workunit.client.0.smithi116.stdout: "application_metadata": {}, 2023-11-02T08:38:39.761 INFO:tasks.workunit.client.0.smithi116.stdout: "read_balance": { 2023-11-02T08:38:39.761 INFO:tasks.workunit.client.0.smithi116.stdout: "score_acting": 2.9900000095367432, 2023-11-02T08:38:39.761 INFO:tasks.workunit.client.0.smithi116.stdout: "score_stable": 2.9900000095367432, 2023-11-02T08:38:39.762 INFO:tasks.workunit.client.0.smithi116.stdout: "optimal_score": 0.67000001668930054, 2023-11-02T08:38:39.762 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_score_acting": 2, 2023-11-02T08:38:39.762 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_score_stable": 2, 2023-11-02T08:38:39.762 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity_weighted": 1, 2023-11-02T08:38:39.762 INFO:tasks.workunit.client.0.smithi116.stdout: "average_primary_affinity": 1, 2023-11-02T08:38:39.763 INFO:tasks.workunit.client.0.smithi116.stdout: "average_primary_affinity_weighted": 1 2023-11-02T08:38:39.763 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.763 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.763 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:38:39.763 INFO:tasks.workunit.client.0.smithi116.stdout: "osds": [ 2023-11-02T08:38:39.764 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.765 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:38:39.765 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "b4aaa3b4-59f6-4f52-81b9-50100a210eb6", 2023-11-02T08:38:39.766 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 1, 2023-11-02T08:38:39.766 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:38:39.766 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:38:39.766 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:38:39.766 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 5, 2023-11-02T08:38:39.767 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 29, 2023-11-02T08:38:39.767 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 36, 2023-11-02T08:38:39.767 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 41, 2023-11-02T08:38:39.767 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 30, 2023-11-02T08:38:39.767 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:38:39.768 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:38:39.768 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:38:39.768 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.768 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:38:39.768 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6802", 2023-11-02T08:38:39.769 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106338 2023-11-02T08:38:39.769 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.769 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.769 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:38:39.769 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6803", 2023-11-02T08:38:39.769 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106338 2023-11-02T08:38:39.770 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.770 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.770 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.770 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:38:39.770 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:38:39.771 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.771 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:38:39.771 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6804", 2023-11-02T08:38:39.771 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106338 2023-11-02T08:38:39.771 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.772 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.772 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:38:39.772 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6805", 2023-11-02T08:38:39.772 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106338 2023-11-02T08:38:39.772 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.773 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.773 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.773 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:38:39.773 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:38:39.773 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.774 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:38:39.774 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6808", 2023-11-02T08:38:39.774 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106338 2023-11-02T08:38:39.774 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.774 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.775 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:38:39.775 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6809", 2023-11-02T08:38:39.775 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106338 2023-11-02T08:38:39.775 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.775 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.775 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.776 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:38:39.776 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:38:39.776 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.776 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:38:39.776 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6806", 2023-11-02T08:38:39.777 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106338 2023-11-02T08:38:39.777 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.777 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.777 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:38:39.778 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6807", 2023-11-02T08:38:39.778 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106338 2023-11-02T08:38:39.778 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.778 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.778 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.779 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6803/106338", 2023-11-02T08:38:39.779 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6805/106338", 2023-11-02T08:38:39.779 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6809/106338", 2023-11-02T08:38:39.779 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6807/106338", 2023-11-02T08:38:39.779 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:38:39.780 INFO:tasks.workunit.client.0.smithi116.stdout: "exists", 2023-11-02T08:38:39.780 INFO:tasks.workunit.client.0.smithi116.stdout: "up" 2023-11-02T08:38:39.780 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.780 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.780 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.781 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:38:39.781 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "3aac1df4-eb52-4ae6-947d-2e4756e32297", 2023-11-02T08:38:39.781 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 0, 2023-11-02T08:38:39.781 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:38:39.782 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:38:39.782 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:38:39.782 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 0, 2023-11-02T08:38:39.782 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 0, 2023-11-02T08:38:39.782 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 10, 2023-11-02T08:38:39.783 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 23, 2023-11-02T08:38:39.783 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 31, 2023-11-02T08:38:39.783 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:38:39.783 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:38:39.783 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:38:39.783 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.784 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:38:39.784 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6810", 2023-11-02T08:38:39.784 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 100309 2023-11-02T08:38:39.784 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.785 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.785 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:38:39.785 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6811", 2023-11-02T08:38:39.785 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 100309 2023-11-02T08:38:39.785 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.785 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.786 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.786 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:38:39.786 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:38:39.786 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.786 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:38:39.787 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6812", 2023-11-02T08:38:39.787 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 100309 2023-11-02T08:38:39.787 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.787 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.787 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:38:39.788 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6813", 2023-11-02T08:38:39.788 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 100309 2023-11-02T08:38:39.788 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.788 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.788 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.789 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:38:39.789 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:38:39.789 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.789 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:38:39.789 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6816", 2023-11-02T08:38:39.790 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 100309 2023-11-02T08:38:39.790 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.790 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.790 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:38:39.790 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6817", 2023-11-02T08:38:39.791 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 100309 2023-11-02T08:38:39.791 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.791 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.791 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.791 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:38:39.791 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:38:39.792 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.792 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:38:39.792 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6814", 2023-11-02T08:38:39.792 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 100309 2023-11-02T08:38:39.792 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.793 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.793 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:38:39.793 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6815", 2023-11-02T08:38:39.793 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 100309 2023-11-02T08:38:39.793 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.794 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.794 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.794 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6811/100309", 2023-11-02T08:38:39.794 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6813/100309", 2023-11-02T08:38:39.794 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6817/100309", 2023-11-02T08:38:39.795 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6815/100309", 2023-11-02T08:38:39.795 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:38:39.795 INFO:tasks.workunit.client.0.smithi116.stdout: "exists" 2023-11-02T08:38:39.795 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.795 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.796 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.796 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:38:39.796 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "d98f6933-2821-4359-90f4-f9b40e2a8e71", 2023-11-02T08:38:39.796 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 1, 2023-11-02T08:38:39.796 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:38:39.797 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:38:39.797 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:38:39.797 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 15, 2023-11-02T08:38:39.797 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 29, 2023-11-02T08:38:39.797 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 38, 2023-11-02T08:38:39.798 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 0, 2023-11-02T08:38:39.798 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 32, 2023-11-02T08:38:39.798 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:38:39.798 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:38:39.798 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:38:39.798 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.799 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:38:39.799 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6810", 2023-11-02T08:38:39.799 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106908 2023-11-02T08:38:39.799 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.799 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.800 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:38:39.800 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6811", 2023-11-02T08:38:39.800 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106908 2023-11-02T08:38:39.800 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.800 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.801 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.801 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:38:39.801 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:38:39.801 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.801 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:38:39.802 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6812", 2023-11-02T08:38:39.802 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106908 2023-11-02T08:38:39.802 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.802 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.802 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:38:39.803 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6813", 2023-11-02T08:38:39.803 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106908 2023-11-02T08:38:39.803 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.803 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.803 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.804 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:38:39.804 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:38:39.804 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.804 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:38:39.804 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6816", 2023-11-02T08:38:39.805 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106908 2023-11-02T08:38:39.805 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.805 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.805 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:38:39.805 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6817", 2023-11-02T08:38:39.805 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106908 2023-11-02T08:38:39.806 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.806 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.806 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.806 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:38:39.806 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:38:39.807 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.807 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:38:39.807 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6814", 2023-11-02T08:38:39.807 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106908 2023-11-02T08:38:39.807 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.808 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.808 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:38:39.808 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6815", 2023-11-02T08:38:39.808 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106908 2023-11-02T08:38:39.808 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.809 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.809 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.809 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6811/106908", 2023-11-02T08:38:39.809 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6813/106908", 2023-11-02T08:38:39.809 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6817/106908", 2023-11-02T08:38:39.810 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6815/106908", 2023-11-02T08:38:39.810 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:38:39.810 INFO:tasks.workunit.client.0.smithi116.stdout: "exists", 2023-11-02T08:38:39.810 INFO:tasks.workunit.client.0.smithi116.stdout: "up" 2023-11-02T08:38:39.810 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.811 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.811 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:38:39.811 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_xinfo": [ 2023-11-02T08:38:39.811 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.811 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:38:39.811 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:37:50.470440+0000", 2023-11-02T08:38:39.812 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:38:39.812 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:38:39.812 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:38:39.812 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:38:39.812 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:35:56.521167+0000", 2023-11-02T08:38:39.813 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:38:39.813 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.813 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.813 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:38:39.813 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:37:51.046017+0000", 2023-11-02T08:38:39.814 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:38:39.814 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:38:39.814 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:38:39.814 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:38:39.814 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:36:29.381154+0000", 2023-11-02T08:38:39.815 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:38:39.815 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.815 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.815 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:38:39.815 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:37:51.569072+0000", 2023-11-02T08:38:39.816 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:38:39.816 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:38:39.816 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:38:39.816 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:38:39.816 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:37:02.597739+0000", 2023-11-02T08:38:39.817 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:38:39.817 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.817 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:38:39.817 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap": [], 2023-11-02T08:38:39.817 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap_items": [], 2023-11-02T08:38:39.817 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap_primaries": [], 2023-11-02T08:38:39.818 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_temp": [], 2023-11-02T08:38:39.818 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_temp": [], 2023-11-02T08:38:39.818 INFO:tasks.workunit.client.0.smithi116.stdout: "blocklist": {}, 2023-11-02T08:38:39.818 INFO:tasks.workunit.client.0.smithi116.stdout: "range_blocklist": {}, 2023-11-02T08:38:39.818 INFO:tasks.workunit.client.0.smithi116.stdout: "erasure_code_profiles": { 2023-11-02T08:38:39.819 INFO:tasks.workunit.client.0.smithi116.stdout: "default": { 2023-11-02T08:38:39.819 INFO:tasks.workunit.client.0.smithi116.stdout: "k": "2", 2023-11-02T08:38:39.819 INFO:tasks.workunit.client.0.smithi116.stdout: "m": "2", 2023-11-02T08:38:39.819 INFO:tasks.workunit.client.0.smithi116.stdout: "plugin": "jerasure", 2023-11-02T08:38:39.819 INFO:tasks.workunit.client.0.smithi116.stdout: "technique": "reed_sol_van" 2023-11-02T08:38:39.820 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.820 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.820 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_snaps_queue": [], 2023-11-02T08:38:39.820 INFO:tasks.workunit.client.0.smithi116.stdout: "new_removed_snaps": [], 2023-11-02T08:38:39.820 INFO:tasks.workunit.client.0.smithi116.stdout: "new_purged_snaps": [], 2023-11-02T08:38:39.821 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_node_flags": {}, 2023-11-02T08:38:39.821 INFO:tasks.workunit.client.0.smithi116.stdout: "device_class_flags": {}, 2023-11-02T08:38:39.821 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode": { 2023-11-02T08:38:39.821 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode_enabled": false, 2023-11-02T08:38:39.821 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_bucket_count": 0, 2023-11-02T08:38:39.822 INFO:tasks.workunit.client.0.smithi116.stdout: "degraded_stretch_mode": 0, 2023-11-02T08:38:39.822 INFO:tasks.workunit.client.0.smithi116.stdout: "recovering_stretch_mode": 0, 2023-11-02T08:38:39.822 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode_bucket": 0 2023-11-02T08:38:39.822 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.822 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.823 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_metadata": [ 2023-11-02T08:38:39.823 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.823 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:38:39.823 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:38:39.823 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6804/106338,v1:127.0.0.1:6805/106338]", 2023-11-02T08:38:39.824 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:38:39.824 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:38:39.824 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:38:39.824 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:38:39.824 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:38:39.824 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:38:39.825 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:38:39.825 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:38:39.825 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:38:39.825 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:38:39.825 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2023-11-02T08:38:39.826 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:38:39.826 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:38:39.826 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:38:39.826 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:38:39.826 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:38:39.827 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:38:39.827 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:38:39.827 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:38:39.827 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:38:39.827 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:38:39.828 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:35:31.038465Z", 2023-11-02T08:38:39.828 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:38:39.828 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:38:39.828 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:38:39.828 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:38:39.829 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:38:39.829 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:38:39.829 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:38:39.829 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6802/106338,v1:127.0.0.1:6803/106338]", 2023-11-02T08:38:39.829 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:38:39.830 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/106338,v1:127.0.0.1:6809/106338]", 2023-11-02T08:38:39.830 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/106338,v1:127.0.0.1:6807/106338]", 2023-11-02T08:38:39.830 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:38:39.830 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:38:39.830 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:38:39.831 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:38:39.831 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:38:39.831 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:38:39.831 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:38:39.831 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:38:39.832 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:38:39.832 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/0", 2023-11-02T08:38:39.832 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:38:39.832 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:38:39.832 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:38:39.832 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.833 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.833 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:38:39.833 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:38:39.833 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6812/100309,v1:127.0.0.1:6813/100309]", 2023-11-02T08:38:39.833 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:38:39.834 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:38:39.834 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:38:39.834 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:38:39.834 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:38:39.834 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:38:39.835 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:38:39.835 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:38:39.835 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:38:39.835 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:38:39.835 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2023-11-02T08:38:39.836 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:38:39.836 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:38:39.836 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:38:39.836 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:38:39.836 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:38:39.837 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:38:39.837 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:38:39.837 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:38:39.837 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:38:39.837 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:38:39.837 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:36:02.798675Z", 2023-11-02T08:38:39.838 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:38:39.838 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:38:39.838 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:38:39.838 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:38:39.839 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:38:39.839 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:38:39.839 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:38:39.839 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6810/100309,v1:127.0.0.1:6811/100309]", 2023-11-02T08:38:39.839 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:38:39.839 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/100309,v1:127.0.0.1:6817/100309]", 2023-11-02T08:38:39.840 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/100309,v1:127.0.0.1:6815/100309]", 2023-11-02T08:38:39.840 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:38:39.840 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:38:39.840 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:38:39.840 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:38:39.841 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:38:39.841 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:38:39.841 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:38:39.841 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:38:39.841 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:38:39.842 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/1", 2023-11-02T08:38:39.842 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:38:39.842 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:38:39.842 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:38:39.842 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.843 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.843 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:38:39.843 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:38:39.843 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6812/106908,v1:127.0.0.1:6813/106908]", 2023-11-02T08:38:39.843 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:38:39.844 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:38:39.844 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:38:39.844 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:38:39.844 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:38:39.844 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:38:39.844 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:38:39.845 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:38:39.845 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:38:39.845 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:38:39.845 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2023-11-02T08:38:39.845 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:38:39.846 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:38:39.846 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:38:39.846 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:38:39.846 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:38:39.846 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:38:39.847 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:38:39.847 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:38:39.847 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:38:39.847 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:38:39.847 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:36:36.600371Z", 2023-11-02T08:38:39.848 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:38:39.848 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:38:39.848 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:38:39.848 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:38:39.848 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:38:39.849 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:38:39.849 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:38:39.849 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6810/106908,v1:127.0.0.1:6811/106908]", 2023-11-02T08:38:39.849 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:38:39.849 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/106908,v1:127.0.0.1:6817/106908]", 2023-11-02T08:38:39.850 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/106908,v1:127.0.0.1:6815/106908]", 2023-11-02T08:38:39.850 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:38:39.850 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:38:39.850 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:38:39.850 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:38:39.851 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:38:39.851 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:38:39.851 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:38:39.851 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:38:39.852 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:38:39.852 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/2", 2023-11-02T08:38:39.852 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:38:39.852 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:38:39.852 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:38:39.853 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.853 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:38:39.853 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_clean_epochs": { 2023-11-02T08:38:39.853 INFO:tasks.workunit.client.0.smithi116.stdout: "min_last_epoch_clean": 0, 2023-11-02T08:38:39.853 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_clean": { 2023-11-02T08:38:39.854 INFO:tasks.workunit.client.0.smithi116.stdout: "per_pool": [] 2023-11-02T08:38:39.854 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.854 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_epochs": [ 2023-11-02T08:38:39.854 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.854 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:38:39.855 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 36 2023-11-02T08:38:39.855 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.855 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.855 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:38:39.855 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 10 2023-11-02T08:38:39.856 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.856 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.856 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:38:39.856 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 39 2023-11-02T08:38:39.856 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.856 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.857 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.857 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_first_committed": 1, 2023-11-02T08:38:39.857 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_last_committed": 42, 2023-11-02T08:38:39.857 INFO:tasks.workunit.client.0.smithi116.stdout: "crushmap": { 2023-11-02T08:38:39.857 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": [ 2023-11-02T08:38:39.858 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.858 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:38:39.858 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.0", 2023-11-02T08:38:39.858 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:38:39.858 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.859 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.859 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:38:39.859 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.1", 2023-11-02T08:38:39.859 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:38:39.859 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.859 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.860 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:38:39.860 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.2", 2023-11-02T08:38:39.860 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:38:39.860 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.860 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:38:39.861 INFO:tasks.workunit.client.0.smithi116.stdout: "types": [ 2023-11-02T08:38:39.861 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.861 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 0, 2023-11-02T08:38:39.861 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd" 2023-11-02T08:38:39.861 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.862 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.862 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:38:39.862 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "host" 2023-11-02T08:38:39.862 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.862 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.863 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 2, 2023-11-02T08:38:39.863 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "chassis" 2023-11-02T08:38:39.863 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.863 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.863 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 3, 2023-11-02T08:38:39.864 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "rack" 2023-11-02T08:38:39.864 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.864 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.864 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 4, 2023-11-02T08:38:39.864 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "row" 2023-11-02T08:38:39.864 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.865 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.865 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 5, 2023-11-02T08:38:39.865 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "pdu" 2023-11-02T08:38:39.865 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.866 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.866 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 6, 2023-11-02T08:38:39.866 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "pod" 2023-11-02T08:38:39.866 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.866 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.867 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 7, 2023-11-02T08:38:39.867 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "room" 2023-11-02T08:38:39.867 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.867 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.868 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 8, 2023-11-02T08:38:39.868 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "datacenter" 2023-11-02T08:38:39.868 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.868 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.868 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 9, 2023-11-02T08:38:39.868 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "zone" 2023-11-02T08:38:39.869 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.869 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.869 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 10, 2023-11-02T08:38:39.869 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "region" 2023-11-02T08:38:39.870 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.870 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.870 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:38:39.870 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "root" 2023-11-02T08:38:39.870 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.871 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:38:39.871 INFO:tasks.workunit.client.0.smithi116.stdout: "buckets": [ 2023-11-02T08:38:39.871 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.871 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -1, 2023-11-02T08:38:39.871 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "default", 2023-11-02T08:38:39.872 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:38:39.872 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "root", 2023-11-02T08:38:39.872 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:38:39.872 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:38:39.872 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:38:39.873 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:38:39.873 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.873 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -3, 2023-11-02T08:38:39.873 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:38:39.874 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:38:39.874 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.874 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.874 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.874 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.875 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -2, 2023-11-02T08:38:39.875 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "default~hdd", 2023-11-02T08:38:39.875 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:38:39.875 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "root", 2023-11-02T08:38:39.876 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:38:39.876 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:38:39.876 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:38:39.876 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:38:39.876 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.876 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -4, 2023-11-02T08:38:39.877 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:38:39.877 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:38:39.877 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.877 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.877 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.878 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.878 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -3, 2023-11-02T08:38:39.878 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "smithi116", 2023-11-02T08:38:39.878 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:38:39.878 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "host", 2023-11-02T08:38:39.879 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:38:39.879 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:38:39.879 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:38:39.879 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:38:39.879 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.880 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:38:39.880 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:38:39.880 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:38:39.880 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.880 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.881 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:38:39.881 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:38:39.881 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 1 2023-11-02T08:38:39.881 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.881 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.882 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:38:39.882 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:38:39.882 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 2 2023-11-02T08:38:39.882 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.882 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.883 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.883 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.883 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -4, 2023-11-02T08:38:39.883 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "smithi116~hdd", 2023-11-02T08:38:39.883 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:38:39.884 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "host", 2023-11-02T08:38:39.884 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:38:39.884 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:38:39.884 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:38:39.884 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:38:39.885 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.885 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:38:39.885 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:38:39.885 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:38:39.885 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.885 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.886 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:38:39.886 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:38:39.886 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 1 2023-11-02T08:38:39.886 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.886 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.887 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:38:39.887 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:38:39.887 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 2 2023-11-02T08:38:39.887 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.887 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.888 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.888 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:38:39.888 INFO:tasks.workunit.client.0.smithi116.stdout: "rules": [ 2023-11-02T08:38:39.888 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.888 INFO:tasks.workunit.client.0.smithi116.stdout: "rule_id": 0, 2023-11-02T08:38:39.888 INFO:tasks.workunit.client.0.smithi116.stdout: "rule_name": "replicated_rule", 2023-11-02T08:38:39.889 INFO:tasks.workunit.client.0.smithi116.stdout: "type": 1, 2023-11-02T08:38:39.889 INFO:tasks.workunit.client.0.smithi116.stdout: "steps": [ 2023-11-02T08:38:39.889 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.889 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "take", 2023-11-02T08:38:39.889 INFO:tasks.workunit.client.0.smithi116.stdout: "item": -1, 2023-11-02T08:38:39.890 INFO:tasks.workunit.client.0.smithi116.stdout: "item_name": "default" 2023-11-02T08:38:39.890 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.890 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.890 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "choose_firstn", 2023-11-02T08:38:39.890 INFO:tasks.workunit.client.0.smithi116.stdout: "num": 0, 2023-11-02T08:38:39.891 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "osd" 2023-11-02T08:38:39.891 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.891 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.891 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "emit" 2023-11-02T08:38:39.891 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.892 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:38:39.892 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.892 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:38:39.892 INFO:tasks.workunit.client.0.smithi116.stdout: "tunables": { 2023-11-02T08:38:39.892 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_local_tries": 0, 2023-11-02T08:38:39.892 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_local_fallback_tries": 0, 2023-11-02T08:38:39.893 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_total_tries": 50, 2023-11-02T08:38:39.893 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_descend_once": 1, 2023-11-02T08:38:39.893 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_vary_r": 1, 2023-11-02T08:38:39.893 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_stable": 1, 2023-11-02T08:38:39.893 INFO:tasks.workunit.client.0.smithi116.stdout: "straw_calc_version": 1, 2023-11-02T08:38:39.894 INFO:tasks.workunit.client.0.smithi116.stdout: "allowed_bucket_algs": 54, 2023-11-02T08:38:39.894 INFO:tasks.workunit.client.0.smithi116.stdout: "profile": "jewel", 2023-11-02T08:38:39.894 INFO:tasks.workunit.client.0.smithi116.stdout: "optimal_tunables": 1, 2023-11-02T08:38:39.894 INFO:tasks.workunit.client.0.smithi116.stdout: "legacy_tunables": 0, 2023-11-02T08:38:39.894 INFO:tasks.workunit.client.0.smithi116.stdout: "minimum_required_version": "jewel", 2023-11-02T08:38:39.895 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables": 1, 2023-11-02T08:38:39.895 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables2": 1, 2023-11-02T08:38:39.895 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v2_rules": 0, 2023-11-02T08:38:39.895 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables3": 1, 2023-11-02T08:38:39.895 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v3_rules": 0, 2023-11-02T08:38:39.896 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v4_buckets": 1, 2023-11-02T08:38:39.896 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables5": 1, 2023-11-02T08:38:39.896 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v5_rules": 0 2023-11-02T08:38:39.896 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.896 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_args": {} 2023-11-02T08:38:39.896 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.897 INFO:tasks.workunit.client.0.smithi116.stdout: "fsmap": { 2023-11-02T08:38:39.897 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 1, 2023-11-02T08:38:39.897 INFO:tasks.workunit.client.0.smithi116.stdout: "default_fscid": -1, 2023-11-02T08:38:39.897 INFO:tasks.workunit.client.0.smithi116.stdout: "compat": { 2023-11-02T08:38:39.897 INFO:tasks.workunit.client.0.smithi116.stdout: "compat": {}, 2023-11-02T08:38:39.898 INFO:tasks.workunit.client.0.smithi116.stdout: "ro_compat": {}, 2023-11-02T08:38:39.898 INFO:tasks.workunit.client.0.smithi116.stdout: "incompat": { 2023-11-02T08:38:39.898 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_1": "base v0.20", 2023-11-02T08:38:39.898 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_2": "client writeable ranges", 2023-11-02T08:38:39.898 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_3": "default file layouts on dirs", 2023-11-02T08:38:39.899 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_4": "dir inode in separate object", 2023-11-02T08:38:39.899 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_5": "mds uses versioned encoding", 2023-11-02T08:38:39.899 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_6": "dirfrag is stored in omap", 2023-11-02T08:38:39.899 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_8": "no anchor table", 2023-11-02T08:38:39.899 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_9": "file layout v2", 2023-11-02T08:38:39.900 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_10": "snaprealm v2" 2023-11-02T08:38:39.900 INFO:tasks.workunit.client.0.smithi116.stderr:report 2201262966 2023-11-02T08:38:39.901 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.901 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.901 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_flags": { 2023-11-02T08:38:39.901 INFO:tasks.workunit.client.0.smithi116.stdout: "enable_multiple": true, 2023-11-02T08:38:39.901 INFO:tasks.workunit.client.0.smithi116.stdout: "ever_enabled_multiple": true 2023-11-02T08:38:39.902 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.902 INFO:tasks.workunit.client.0.smithi116.stdout: "standbys": [], 2023-11-02T08:38:39.902 INFO:tasks.workunit.client.0.smithi116.stdout: "filesystems": [] 2023-11-02T08:38:39.902 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.903 INFO:tasks.workunit.client.0.smithi116.stdout: "mdsmap_first_committed": 1, 2023-11-02T08:38:39.903 INFO:tasks.workunit.client.0.smithi116.stdout: "mdsmap_last_committed": 1, 2023-11-02T08:38:39.903 INFO:tasks.workunit.client.0.smithi116.stdout: "auth": { 2023-11-02T08:38:39.903 INFO:tasks.workunit.client.0.smithi116.stdout: "first_committed": 1, 2023-11-02T08:38:39.903 INFO:tasks.workunit.client.0.smithi116.stdout: "last_committed": 5, 2023-11-02T08:38:39.904 INFO:tasks.workunit.client.0.smithi116.stdout: "num_secrets": 3 2023-11-02T08:38:39.904 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.904 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg": 1, 2023-11-02T08:38:39.904 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_active": 1, 2023-11-02T08:38:39.904 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_unknown": 0, 2023-11-02T08:38:39.905 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osd": 3, 2023-11-02T08:38:39.905 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_sum": { 2023-11-02T08:38:39.905 INFO:tasks.workunit.client.0.smithi116.stdout: "stat_sum": { 2023-11-02T08:38:39.905 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes": 48300, 2023-11-02T08:38:39.905 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects": 100, 2023-11-02T08:38:39.906 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_clones": 0, 2023-11-02T08:38:39.906 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_copies": 300, 2023-11-02T08:38:39.906 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing_on_primary": 0, 2023-11-02T08:38:39.906 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing": 0, 2023-11-02T08:38:39.906 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_degraded": 100, 2023-11-02T08:38:39.907 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_misplaced": 0, 2023-11-02T08:38:39.907 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_unfound": 0, 2023-11-02T08:38:39.907 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_dirty": 100, 2023-11-02T08:38:39.907 INFO:tasks.workunit.client.0.smithi116.stdout: "num_whiteouts": 0, 2023-11-02T08:38:39.907 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read": 0, 2023-11-02T08:38:39.908 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read_kb": 0, 2023-11-02T08:38:39.908 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write": 100, 2023-11-02T08:38:39.908 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write_kb": 100, 2023-11-02T08:38:39.908 INFO:tasks.workunit.client.0.smithi116.stdout: "num_scrub_errors": 0, 2023-11-02T08:38:39.908 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shallow_scrub_errors": 0, 2023-11-02T08:38:39.909 INFO:tasks.workunit.client.0.smithi116.stdout: "num_deep_scrub_errors": 0, 2023-11-02T08:38:39.909 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_recovered": 0, 2023-11-02T08:38:39.909 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_recovered": 0, 2023-11-02T08:38:39.909 INFO:tasks.workunit.client.0.smithi116.stdout: "num_keys_recovered": 0, 2023-11-02T08:38:39.909 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_omap": 0, 2023-11-02T08:38:39.910 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_hit_set_archive": 0, 2023-11-02T08:38:39.910 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_hit_set_archive": 0, 2023-11-02T08:38:39.910 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush": 0, 2023-11-02T08:38:39.910 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_kb": 0, 2023-11-02T08:38:39.910 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict": 0, 2023-11-02T08:38:39.911 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_kb": 0, 2023-11-02T08:38:39.911 INFO:tasks.workunit.client.0.smithi116.stdout: "num_promote": 0, 2023-11-02T08:38:39.911 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_high": 0, 2023-11-02T08:38:39.911 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_low": 0, 2023-11-02T08:38:39.911 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_some": 0, 2023-11-02T08:38:39.912 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_full": 0, 2023-11-02T08:38:39.912 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_pinned": 0, 2023-11-02T08:38:39.912 INFO:tasks.workunit.client.0.smithi116.stdout: "num_legacy_snapsets": 0, 2023-11-02T08:38:39.912 INFO:tasks.workunit.client.0.smithi116.stdout: "num_large_omap_objects": 0, 2023-11-02T08:38:39.913 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_manifest": 0, 2023-11-02T08:38:39.913 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_bytes": 0, 2023-11-02T08:38:39.913 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_keys": 0, 2023-11-02T08:38:39.913 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_repaired": 0 2023-11-02T08:38:39.913 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.914 INFO:tasks.workunit.client.0.smithi116.stdout: "store_stats": { 2023-11-02T08:38:39.914 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 0, 2023-11-02T08:38:39.914 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 0, 2023-11-02T08:38:39.914 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:38:39.915 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 0, 2023-11-02T08:38:39.915 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 0, 2023-11-02T08:38:39.915 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:38:39.915 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:38:39.916 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:38:39.916 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 0, 2023-11-02T08:38:39.916 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 0 2023-11-02T08:38:39.916 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.916 INFO:tasks.workunit.client.0.smithi116.stdout: "log_size": 16, 2023-11-02T08:38:39.917 INFO:tasks.workunit.client.0.smithi116.stdout: "ondisk_log_size": 16, 2023-11-02T08:38:39.917 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 2, 2023-11-02T08:38:39.917 INFO:tasks.workunit.client.0.smithi116.stdout: "acting": 2, 2023-11-02T08:38:39.917 INFO:tasks.workunit.client.0.smithi116.stdout: "num_store_stats": 0 2023-11-02T08:38:39.917 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.917 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_sum": { 2023-11-02T08:38:39.918 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 0, 2023-11-02T08:38:39.918 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 0, 2023-11-02T08:38:39.918 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pgs": 3, 2023-11-02T08:38:39.918 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osds": 3, 2023-11-02T08:38:39.918 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_osds": 3, 2023-11-02T08:38:39.919 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_omap_osds": 3, 2023-11-02T08:38:39.919 INFO:tasks.workunit.client.0.smithi116.stdout: "kb": 314572800, 2023-11-02T08:38:39.919 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used": 83304, 2023-11-02T08:38:39.919 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_data": 2120, 2023-11-02T08:38:39.919 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_omap": 2, 2023-11-02T08:38:39.920 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_meta": 81149, 2023-11-02T08:38:39.920 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_avail": 314489496, 2023-11-02T08:38:39.920 INFO:tasks.workunit.client.0.smithi116.stdout: "statfs": { 2023-11-02T08:38:39.920 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 322122547200, 2023-11-02T08:38:39.920 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 322037243904, 2023-11-02T08:38:39.921 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:38:39.921 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 2170880, 2023-11-02T08:38:39.921 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 423993, 2023-11-02T08:38:39.921 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:38:39.921 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:38:39.922 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:38:39.922 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 2366, 2023-11-02T08:38:39.922 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 83097282 2023-11-02T08:38:39.922 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.922 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_peers": [], 2023-11-02T08:38:39.923 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_trim_queue_len": 0, 2023-11-02T08:38:39.923 INFO:tasks.workunit.client.0.smithi116.stdout: "num_snap_trimming": 0, 2023-11-02T08:38:39.923 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shards_repaired": 0, 2023-11-02T08:38:39.923 INFO:tasks.workunit.client.0.smithi116.stdout: "op_queue_age_hist": { 2023-11-02T08:38:39.923 INFO:tasks.workunit.client.0.smithi116.stdout: "histogram": [], 2023-11-02T08:38:39.924 INFO:tasks.workunit.client.0.smithi116.stdout: "upper_bound": 1 2023-11-02T08:38:39.924 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.924 INFO:tasks.workunit.client.0.smithi116.stdout: "perf_stat": { 2023-11-02T08:38:39.924 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ms": 0, 2023-11-02T08:38:39.924 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ms": 0, 2023-11-02T08:38:39.925 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ns": 0, 2023-11-02T08:38:39.925 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ns": 0 2023-11-02T08:38:39.925 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.925 INFO:tasks.workunit.client.0.smithi116.stdout: "alerts": [], 2023-11-02T08:38:39.925 INFO:tasks.workunit.client.0.smithi116.stdout: "network_ping_times": [] 2023-11-02T08:38:39.925 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.926 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_sum_by_class": { 2023-11-02T08:38:39.926 INFO:tasks.workunit.client.0.smithi116.stdout: "hdd": { 2023-11-02T08:38:39.926 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 0, 2023-11-02T08:38:39.926 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 0, 2023-11-02T08:38:39.926 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pgs": 3, 2023-11-02T08:38:39.927 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osds": 3, 2023-11-02T08:38:39.927 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_osds": 3, 2023-11-02T08:38:39.927 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_omap_osds": 3, 2023-11-02T08:38:39.927 INFO:tasks.workunit.client.0.smithi116.stdout: "kb": 314572800, 2023-11-02T08:38:39.927 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used": 83304, 2023-11-02T08:38:39.928 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_data": 2120, 2023-11-02T08:38:39.928 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_omap": 2, 2023-11-02T08:38:39.928 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_meta": 81149, 2023-11-02T08:38:39.928 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_avail": 314489496, 2023-11-02T08:38:39.928 INFO:tasks.workunit.client.0.smithi116.stdout: "statfs": { 2023-11-02T08:38:39.929 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 322122547200, 2023-11-02T08:38:39.929 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 322037243904, 2023-11-02T08:38:39.929 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:38:39.929 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 2170880, 2023-11-02T08:38:39.929 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 423993, 2023-11-02T08:38:39.929 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:38:39.930 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:38:39.930 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:38:39.930 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 2366, 2023-11-02T08:38:39.930 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 83097282 2023-11-02T08:38:39.930 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.931 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_peers": [], 2023-11-02T08:38:39.931 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_trim_queue_len": 0, 2023-11-02T08:38:39.931 INFO:tasks.workunit.client.0.smithi116.stdout: "num_snap_trimming": 0, 2023-11-02T08:38:39.931 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shards_repaired": 0, 2023-11-02T08:38:39.931 INFO:tasks.workunit.client.0.smithi116.stdout: "op_queue_age_hist": { 2023-11-02T08:38:39.932 INFO:tasks.workunit.client.0.smithi116.stdout: "histogram": [], 2023-11-02T08:38:39.932 INFO:tasks.workunit.client.0.smithi116.stdout: "upper_bound": 1 2023-11-02T08:38:39.932 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.932 INFO:tasks.workunit.client.0.smithi116.stdout: "perf_stat": { 2023-11-02T08:38:39.932 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ms": 0, 2023-11-02T08:38:39.933 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ms": 0, 2023-11-02T08:38:39.933 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ns": 0, 2023-11-02T08:38:39.933 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ns": 0 2023-11-02T08:38:39.933 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.933 INFO:tasks.workunit.client.0.smithi116.stdout: "alerts": [], 2023-11-02T08:38:39.934 INFO:tasks.workunit.client.0.smithi116.stdout: "network_ping_times": [] 2023-11-02T08:38:39.934 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.934 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.934 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_stats": [ 2023-11-02T08:38:39.934 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.935 INFO:tasks.workunit.client.0.smithi116.stdout: "poolid": 1, 2023-11-02T08:38:39.935 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg": 1, 2023-11-02T08:38:39.935 INFO:tasks.workunit.client.0.smithi116.stdout: "stat_sum": { 2023-11-02T08:38:39.935 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes": 48300, 2023-11-02T08:38:39.935 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects": 100, 2023-11-02T08:38:39.935 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_clones": 0, 2023-11-02T08:38:39.936 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_copies": 300, 2023-11-02T08:38:39.936 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing_on_primary": 0, 2023-11-02T08:38:39.936 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing": 0, 2023-11-02T08:38:39.936 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_degraded": 100, 2023-11-02T08:38:39.936 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_misplaced": 0, 2023-11-02T08:38:39.937 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_unfound": 0, 2023-11-02T08:38:39.937 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_dirty": 100, 2023-11-02T08:38:39.937 INFO:tasks.workunit.client.0.smithi116.stdout: "num_whiteouts": 0, 2023-11-02T08:38:39.937 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read": 0, 2023-11-02T08:38:39.937 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read_kb": 0, 2023-11-02T08:38:39.938 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write": 100, 2023-11-02T08:38:39.938 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write_kb": 100, 2023-11-02T08:38:39.938 INFO:tasks.workunit.client.0.smithi116.stdout: "num_scrub_errors": 0, 2023-11-02T08:38:39.938 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shallow_scrub_errors": 0, 2023-11-02T08:38:39.938 INFO:tasks.workunit.client.0.smithi116.stdout: "num_deep_scrub_errors": 0, 2023-11-02T08:38:39.939 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_recovered": 0, 2023-11-02T08:38:39.939 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_recovered": 0, 2023-11-02T08:38:39.939 INFO:tasks.workunit.client.0.smithi116.stdout: "num_keys_recovered": 0, 2023-11-02T08:38:39.939 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_omap": 0, 2023-11-02T08:38:39.939 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_hit_set_archive": 0, 2023-11-02T08:38:39.940 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_hit_set_archive": 0, 2023-11-02T08:38:39.940 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush": 0, 2023-11-02T08:38:39.940 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_kb": 0, 2023-11-02T08:38:39.940 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict": 0, 2023-11-02T08:38:39.940 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_kb": 0, 2023-11-02T08:38:39.941 INFO:tasks.workunit.client.0.smithi116.stdout: "num_promote": 0, 2023-11-02T08:38:39.941 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_high": 0, 2023-11-02T08:38:39.941 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_low": 0, 2023-11-02T08:38:39.941 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_some": 0, 2023-11-02T08:38:39.941 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_full": 0, 2023-11-02T08:38:39.942 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_pinned": 0, 2023-11-02T08:38:39.942 INFO:tasks.workunit.client.0.smithi116.stdout: "num_legacy_snapsets": 0, 2023-11-02T08:38:39.942 INFO:tasks.workunit.client.0.smithi116.stdout: "num_large_omap_objects": 0, 2023-11-02T08:38:39.942 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_manifest": 0, 2023-11-02T08:38:39.942 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_bytes": 0, 2023-11-02T08:38:39.943 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_keys": 0, 2023-11-02T08:38:39.943 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_repaired": 0 2023-11-02T08:38:39.943 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.943 INFO:tasks.workunit.client.0.smithi116.stdout: "store_stats": { 2023-11-02T08:38:39.943 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 0, 2023-11-02T08:38:39.944 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 0, 2023-11-02T08:38:39.944 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:38:39.944 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 1220608, 2023-11-02T08:38:39.944 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 143228, 2023-11-02T08:38:39.944 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:38:39.945 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:38:39.945 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:38:39.945 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 0, 2023-11-02T08:38:39.945 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 0 2023-11-02T08:38:39.945 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.946 INFO:tasks.workunit.client.0.smithi116.stdout: "log_size": 16, 2023-11-02T08:38:39.946 INFO:tasks.workunit.client.0.smithi116.stdout: "ondisk_log_size": 16, 2023-11-02T08:38:39.946 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 2, 2023-11-02T08:38:39.946 INFO:tasks.workunit.client.0.smithi116.stdout: "acting": 2, 2023-11-02T08:38:39.946 INFO:tasks.workunit.client.0.smithi116.stdout: "num_store_stats": 3 2023-11-02T08:38:39.947 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.947 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:38:39.947 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_stats": [ 2023-11-02T08:38:39.947 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.947 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:38:39.948 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 154618822667 2023-11-02T08:38:39.948 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.948 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.948 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:38:39.948 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 42949672986 2023-11-02T08:38:39.949 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.949 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.949 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:38:39.949 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 163208757258 2023-11-02T08:38:39.949 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.949 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:38:39.950 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_by_state": [ 2023-11-02T08:38:39.950 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.950 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "active+undersized+degraded", 2023-11-02T08:38:39.950 INFO:tasks.workunit.client.0.smithi116.stdout: "num": 1 2023-11-02T08:38:39.950 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.951 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:38:39.951 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_by_osd": [ 2023-11-02T08:38:39.951 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.951 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:38:39.951 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 0, 2023-11-02T08:38:39.952 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 1, 2023-11-02T08:38:39.952 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:38:39.952 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.952 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.952 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:38:39.953 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 1, 2023-11-02T08:38:39.953 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 1, 2023-11-02T08:38:39.953 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:38:39.953 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.953 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.954 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:38:39.954 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 0, 2023-11-02T08:38:39.954 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 0, 2023-11-02T08:38:39.954 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:38:39.954 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.955 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:38:39.955 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snaps": [ 2023-11-02T08:38:39.955 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:38:39.955 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": 1, 2023-11-02T08:38:39.955 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snaps": {} 2023-11-02T08:38:39.955 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.956 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:38:39.956 INFO:tasks.workunit.client.0.smithi116.stdout: "servicemap": { 2023-11-02T08:38:39.956 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 4, 2023-11-02T08:38:39.956 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:37:31.728244+0000", 2023-11-02T08:38:39.956 INFO:tasks.workunit.client.0.smithi116.stdout: "services": { 2023-11-02T08:38:39.957 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": { 2023-11-02T08:38:39.957 INFO:tasks.workunit.client.0.smithi116.stdout: "daemons": { 2023-11-02T08:38:39.957 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": "", 2023-11-02T08:38:39.957 INFO:tasks.workunit.client.0.smithi116.stdout: "2": { 2023-11-02T08:38:39.957 INFO:tasks.workunit.client.0.smithi116.stdout: "start_epoch": 0, 2023-11-02T08:38:39.958 INFO:tasks.workunit.client.0.smithi116.stdout: "start_stamp": "0.000000", 2023-11-02T08:38:39.958 INFO:tasks.workunit.client.0.smithi116.stdout: "gid": 0, 2023-11-02T08:38:39.958 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "(unrecognized address family 0)/0", 2023-11-02T08:38:39.958 INFO:tasks.workunit.client.0.smithi116.stdout: "metadata": {}, 2023-11-02T08:38:39.958 INFO:tasks.workunit.client.0.smithi116.stdout: "task_status": {} 2023-11-02T08:38:39.959 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.959 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.959 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.959 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.959 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:38:39.960 INFO:tasks.workunit.client.0.smithi116.stdout: "mgrstat_first_committed": 1, 2023-11-02T08:38:39.960 INFO:tasks.workunit.client.0.smithi116.stdout: "mgrstat_last_committed": 86, 2023-11-02T08:38:39.960 INFO:tasks.workunit.client.0.smithi116.stdout: "logm_first_committed": 1, 2023-11-02T08:38:39.960 INFO:tasks.workunit.client.0.smithi116.stdout: "logm_last_committed": 366, 2023-11-02T08:38:39.960 INFO:tasks.workunit.client.0.smithi116.stdout: "paxos": { 2023-11-02T08:38:39.960 INFO:tasks.workunit.client.0.smithi116.stdout: "first_committed": 1, 2023-11-02T08:38:39.961 INFO:tasks.workunit.client.0.smithi116.stdout: "last_committed": 536, 2023-11-02T08:38:39.961 INFO:tasks.workunit.client.0.smithi116.stdout: "last_pn": 0, 2023-11-02T08:38:39.961 INFO:tasks.workunit.client.0.smithi116.stdout: "accepted_pn": 0 2023-11-02T08:38:39.961 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:38:39.961 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T08:38:39.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2023-11-02T08:38:39.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2023-11-02T08:38:39.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2023-11-02T08:38:39.963 INFO:tasks.workunit.client.0.smithi116.stdout:writing non-divergent object existing_4 2023-11-02T08:38:39.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2023-11-02T08:38:39.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2023-11-02T08:38:40.130 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:38:40.130 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2023-11-02T08:38:03.562008+0000 29'100 42:124 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:37:06.739239+0000 0'0 2023-11-02T08:37:06.739239+0000 0 0 periodic scrub scheduled @ 2023-11-03T16:37:11.730509+0000 0 0 2023-11-02T08:38:40.130 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:38:40.131 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:38:40.131 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:38:40.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2023-11-02T08:38:40.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2023-11-02T08:38:40.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2023-11-02T08:38:40.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T08:38:40.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T08:38:40.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T08:38:40.224 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2023-11-02T08:38:40.224 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T08:38:40.225 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T08:38:40.225 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T08:38:40.225 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T08:38:40.225 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T08:38:40.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2023-11-02T08:38:40.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T08:38:40.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T08:38:40.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T08:38:40.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:38:40.274 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:38:40.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:38:40.650 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:38:40.650 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:38:40.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:38:40.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:38:40.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:38:40.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=154618822669 2023-11-02T08:38:40.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 154618822669 2023-11-02T08:38:40.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-154618822669' 2023-11-02T08:38:40.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:38:40.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:38:40.855 INFO:tasks.workunit.client.0.smithi116.stderr:Error ENXIO: problem getting command descriptions from osd.1 2023-11-02T08:38:40.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2023-11-02T08:38:40.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2023-11-02T08:38:40.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2023-11-02T08:38:40.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:38:40.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:38:40.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=163208757260 2023-11-02T08:38:40.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 163208757260 2023-11-02T08:38:40.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-154618822669 2-163208757260' 2023-11-02T08:38:40.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:38:40.978 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-154618822669 2023-11-02T08:38:40.978 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:38:40.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:38:40.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-154618822669 2023-11-02T08:38:40.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:38:40.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=154618822669 2023-11-02T08:38:40.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 154618822669' 2023-11-02T08:38:40.980 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 154618822669 2023-11-02T08:38:40.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:38:41.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 154618822667 -lt 154618822669 2023-11-02T08:38:41.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T08:38:42.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T08:38:42.358 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:38:42.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 154618822669 -lt 154618822669 2023-11-02T08:38:42.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:38:42.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-163208757260 2023-11-02T08:38:42.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:38:42.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:38:42.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-163208757260 2023-11-02T08:38:42.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:38:42.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=163208757260 2023-11-02T08:38:42.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 163208757260' 2023-11-02T08:38:42.744 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 163208757260 2023-11-02T08:38:42.745 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:38:43.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 163208757261 -lt 163208757260 2023-11-02T08:38:43.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T08:38:43.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:38:43.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:38:43.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2023-11-02T08:38:43.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:38:43.629 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:38:43.629 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:38:43.630 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:38:43.630 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:38:43.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:38:43.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:38:44.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:38:44.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:38:44.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:38:44.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:38:44.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:38:44.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2023-11-02T08:38:44.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2023-11-02T08:38:44.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2023-11-02T08:38:44.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:38:44.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T08:38:44.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:38:44.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:38:44.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:38:44.629 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:38:44.629 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:38:44.629 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:38:44.629 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:38:44.630 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:38:44.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:38:44.996 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:38:44.997 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:38:44.997 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:38:45.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:38:45.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:38:45.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:38:45.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:38:45.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:38:45.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:38:45.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:38:45.510 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:38:45.510 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:38:46.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:38:46.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:38:46.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2023-11-02T08:38:46.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:38:46.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2023-11-02T08:38:46.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:38:46.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:38:46.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:38:46.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:38:46.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:38:46.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:38:46.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:38:46.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:38:46.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:38:46.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:38:46.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:38:46.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:38:47.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:38:47.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:38:47.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:38:47.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:38:47.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:38:47.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:38:47.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:38:47.133 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:38:47.133 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:38:47.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:38:47.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:38:47.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2023-11-02T08:38:47.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:38:47.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2023-11-02T08:38:48.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:38:48.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:38:48.063 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:38:48.064 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:38:48.064 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:38:48.064 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:38:48.064 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:38:48.064 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:38:48.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:38:48.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:38:48.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:38:48.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:38:48.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:38:48.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:38:48.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:38:48.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:38:48.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:38:48.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:38:48.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:38:48.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:38:48.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:38:49.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:38:49.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:38:49.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2023-11-02T08:38:49.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:38:49.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2023-11-02T08:38:50.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:38:50.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:38:50.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:38:50.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:38:50.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:38:50.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:38:50.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:38:50.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:38:50.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:38:50.657 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:38:50.658 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:38:50.658 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:38:51.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:38:51.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:38:51.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:38:51.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:38:51.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:38:51.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:38:51.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:38:51.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:38:51.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:38:51.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:38:51.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:38:51.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2023-11-02T08:38:51.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:38:51.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2023-11-02T08:38:53.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:38:53.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:38:53.272 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:38:53.272 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:38:53.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:38:53.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:38:53.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:38:53.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:38:53.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:38:53.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:38:53.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:38:53.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:38:54.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:38:54.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:38:54.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:38:54.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:38:54.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:38:54.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:38:54.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:38:54.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:38:54.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:38:54.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:38:54.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:38:54.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2023-11-02T08:38:54.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:38:54.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2023-11-02T08:38:57.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:38:57.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:38:57.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:38:57.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:38:57.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:38:57.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:38:57.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:38:57.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:38:58.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:38:58.321 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:38:58.321 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:38:58.321 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:38:58.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:38:58.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:38:58.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:38:58.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:38:58.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:38:58.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:38:58.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:38:58.825 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:38:58.825 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:38:59.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:38:59.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:38:59.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2023-11-02T08:38:59.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:38:59.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2023-11-02T08:39:05.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:39:05.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:39:05.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:39:05.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:39:05.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:39:05.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:39:05.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:39:05.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:39:06.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:39:06.155 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:39:06.155 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:39:06.156 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:39:06.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:39:06.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:39:06.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:39:06.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:39:06.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:39:06.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:39:06.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:39:06.660 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:39:06.660 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:39:07.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:39:07.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:39:07.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2023-11-02T08:39:07.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:39:07.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2023-11-02T08:39:14.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:39:14.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:39:14.534 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:39:14.534 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:39:14.534 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:39:14.534 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:39:14.534 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:39:14.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:39:14.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:39:14.911 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:39:14.911 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:39:14.911 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:39:15.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:39:15.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:39:15.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:39:15.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:39:15.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:39:15.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:39:15.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:39:15.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:39:15.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:39:15.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:39:15.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:39:15.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2023-11-02T08:39:15.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2023-11-02T08:39:16.449 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T08:39:16.449 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_fingerprint": "d678552a-81a2-4f9f-a139-3b7833870c2a", 2023-11-02T08:39:16.449 INFO:tasks.workunit.client.0.smithi116.stdout: "version": "18.2.0-1159-gb5a9486b", 2023-11-02T08:39:16.449 INFO:tasks.workunit.client.0.smithi116.stdout: "commit": "b5a9486b4a6f144f83c51b7ec33111a2c86dc081", 2023-11-02T08:39:16.450 INFO:tasks.workunit.client.0.smithi116.stdout: "timestamp": "2023-11-02T08:39:16.445432+0000", 2023-11-02T08:39:16.450 INFO:tasks.workunit.client.0.smithi116.stdout: "tag": "", 2023-11-02T08:39:16.450 INFO:tasks.workunit.client.0.smithi116.stdout: "health": { 2023-11-02T08:39:16.450 INFO:tasks.workunit.client.0.smithi116.stdout: "status": "HEALTH_WARN", 2023-11-02T08:39:16.450 INFO:tasks.workunit.client.0.smithi116.stdout: "checks": { 2023-11-02T08:39:16.451 INFO:tasks.workunit.client.0.smithi116.stdout: "OSDMAP_FLAGS": { 2023-11-02T08:39:16.451 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:39:16.451 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:39:16.451 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "nodown,noout,noin flag(s) set", 2023-11-02T08:39:16.451 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 17 2023-11-02T08:39:16.452 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.452 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [], 2023-11-02T08:39:16.452 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:39:16.452 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.452 INFO:tasks.workunit.client.0.smithi116.stdout: "OSD_DOWN": { 2023-11-02T08:39:16.453 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:39:16.453 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:39:16.453 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "1 osds down", 2023-11-02T08:39:16.453 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 1 2023-11-02T08:39:16.454 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.454 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:39:16.455 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.455 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "osd.1 (root=default,host=smithi116) is down" 2023-11-02T08:39:16.457 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.457 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:39:16.457 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:39:16.457 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.458 INFO:tasks.workunit.client.0.smithi116.stdout: "PG_DEGRADED": { 2023-11-02T08:39:16.458 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:39:16.458 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:39:16.458 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2023-11-02T08:39:16.458 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 2 2023-11-02T08:39:16.459 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.459 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:39:16.459 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.459 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.0 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2023-11-02T08:39:16.459 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.459 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:39:16.460 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:39:16.460 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.460 INFO:tasks.workunit.client.0.smithi116.stdout: "POOL_APP_NOT_ENABLED": { 2023-11-02T08:39:16.460 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:39:16.460 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:39:16.461 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "1 pool(s) do not have an application enabled", 2023-11-02T08:39:16.461 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 1 2023-11-02T08:39:16.461 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.461 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:39:16.461 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.462 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "application not enabled on pool 'test'" 2023-11-02T08:39:16.462 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.462 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.462 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2023-11-02T08:39:16.463 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.463 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:39:16.463 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:39:16.463 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.463 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.464 INFO:tasks.workunit.client.0.smithi116.stdout: "mutes": [] 2023-11-02T08:39:16.464 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.464 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap_first_committed": 1, 2023-11-02T08:39:16.464 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap_last_committed": 1, 2023-11-02T08:39:16.464 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap": { 2023-11-02T08:39:16.465 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 1, 2023-11-02T08:39:16.465 INFO:tasks.workunit.client.0.smithi116.stdout: "fsid": "0c9a372f-cec7-47d9-a309-8248453aabeb", 2023-11-02T08:39:16.466 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:35:24.297501Z", 2023-11-02T08:39:16.466 INFO:tasks.workunit.client.0.smithi116.stdout: "created": "2023-11-02T08:35:24.297501Z", 2023-11-02T08:39:16.466 INFO:tasks.workunit.client.0.smithi116.stdout: "min_mon_release": 18, 2023-11-02T08:39:16.466 INFO:tasks.workunit.client.0.smithi116.stdout: "min_mon_release_name": "reef", 2023-11-02T08:39:16.466 INFO:tasks.workunit.client.0.smithi116.stdout: "election_strategy": 1, 2023-11-02T08:39:16.467 INFO:tasks.workunit.client.0.smithi116.stdout: "disallowed_leaders: ": "", 2023-11-02T08:39:16.467 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode": false, 2023-11-02T08:39:16.467 INFO:tasks.workunit.client.0.smithi116.stdout: "tiebreaker_mon": "", 2023-11-02T08:39:16.467 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_ranks: ": "", 2023-11-02T08:39:16.467 INFO:tasks.workunit.client.0.smithi116.stdout: "features": { 2023-11-02T08:39:16.468 INFO:tasks.workunit.client.0.smithi116.stdout: "persistent": [ 2023-11-02T08:39:16.468 INFO:tasks.workunit.client.0.smithi116.stdout: "kraken", 2023-11-02T08:39:16.468 INFO:tasks.workunit.client.0.smithi116.stdout: "luminous", 2023-11-02T08:39:16.468 INFO:tasks.workunit.client.0.smithi116.stdout: "mimic", 2023-11-02T08:39:16.469 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap-prune", 2023-11-02T08:39:16.469 INFO:tasks.workunit.client.0.smithi116.stdout: "nautilus", 2023-11-02T08:39:16.469 INFO:tasks.workunit.client.0.smithi116.stdout: "octopus", 2023-11-02T08:39:16.469 INFO:tasks.workunit.client.0.smithi116.stdout: "pacific", 2023-11-02T08:39:16.470 INFO:tasks.workunit.client.0.smithi116.stdout: "elector-pinging", 2023-11-02T08:39:16.470 INFO:tasks.workunit.client.0.smithi116.stdout: "quincy", 2023-11-02T08:39:16.470 INFO:tasks.workunit.client.0.smithi116.stdout: "reef" 2023-11-02T08:39:16.470 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:39:16.471 INFO:tasks.workunit.client.0.smithi116.stdout: "optional": [] 2023-11-02T08:39:16.471 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.471 INFO:tasks.workunit.client.0.smithi116.stdout: "mons": [ 2023-11-02T08:39:16.471 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.471 INFO:tasks.workunit.client.0.smithi116.stdout: "rank": 0, 2023-11-02T08:39:16.471 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "a", 2023-11-02T08:39:16.472 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:39:16.472 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:39:16.472 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.472 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:39:16.472 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:7115", 2023-11-02T08:39:16.473 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 0 2023-11-02T08:39:16.473 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.473 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.473 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.473 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:7115/0", 2023-11-02T08:39:16.474 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:7115/0", 2023-11-02T08:39:16.474 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 0, 2023-11-02T08:39:16.474 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 0, 2023-11-02T08:39:16.474 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_location": "{}" 2023-11-02T08:39:16.474 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.475 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.475 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.475 INFO:tasks.workunit.client.0.smithi116.stdout: "quorum": [ 2023-11-02T08:39:16.475 INFO:tasks.workunit.client.0.smithi116.stdout: 0 2023-11-02T08:39:16.475 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:39:16.476 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap": { 2023-11-02T08:39:16.476 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 42, 2023-11-02T08:39:16.476 INFO:tasks.workunit.client.0.smithi116.stdout: "fsid": "0c9a372f-cec7-47d9-a309-8248453aabeb", 2023-11-02T08:39:16.476 INFO:tasks.workunit.client.0.smithi116.stdout: "created": "2023-11-02T08:35:25.095191+0000", 2023-11-02T08:39:16.476 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:38:03.312654+0000", 2023-11-02T08:39:16.476 INFO:tasks.workunit.client.0.smithi116.stdout: "last_up_change": "2023-11-02T08:38:01.787977+0000", 2023-11-02T08:39:16.477 INFO:tasks.workunit.client.0.smithi116.stdout: "last_in_change": "2023-11-02T08:38:03.008439+0000", 2023-11-02T08:39:16.477 INFO:tasks.workunit.client.0.smithi116.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-11-02T08:39:16.477 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_num": 5800384, 2023-11-02T08:39:16.477 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_set": [ 2023-11-02T08:39:16.477 INFO:tasks.workunit.client.0.smithi116.stdout: "nodown", 2023-11-02T08:39:16.478 INFO:tasks.workunit.client.0.smithi116.stdout: "noin", 2023-11-02T08:39:16.478 INFO:tasks.workunit.client.0.smithi116.stdout: "noout", 2023-11-02T08:39:16.478 INFO:tasks.workunit.client.0.smithi116.stdout: "pglog_hardlimit", 2023-11-02T08:39:16.478 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snapdirs", 2023-11-02T08:39:16.478 INFO:tasks.workunit.client.0.smithi116.stdout: "recovery_deletes", 2023-11-02T08:39:16.479 INFO:tasks.workunit.client.0.smithi116.stdout: "sortbitwise" 2023-11-02T08:39:16.479 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:39:16.479 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_version": 9, 2023-11-02T08:39:16.479 INFO:tasks.workunit.client.0.smithi116.stdout: "full_ratio": 0.99000000953674316, 2023-11-02T08:39:16.479 INFO:tasks.workunit.client.0.smithi116.stdout: "backfillfull_ratio": 0.99000000953674316, 2023-11-02T08:39:16.480 INFO:tasks.workunit.client.0.smithi116.stdout: "nearfull_ratio": 0.85000002384185791, 2023-11-02T08:39:16.480 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_snapshot": "", 2023-11-02T08:39:16.480 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_max": 1, 2023-11-02T08:39:16.480 INFO:tasks.workunit.client.0.smithi116.stdout: "max_osd": 3, 2023-11-02T08:39:16.480 INFO:tasks.workunit.client.0.smithi116.stdout: "require_min_compat_client": "luminous", 2023-11-02T08:39:16.481 INFO:tasks.workunit.client.0.smithi116.stdout: "min_compat_client": "jewel", 2023-11-02T08:39:16.481 INFO:tasks.workunit.client.0.smithi116.stdout: "require_osd_release": "reef", 2023-11-02T08:39:16.481 INFO:tasks.workunit.client.0.smithi116.stdout: "allow_crimson": false, 2023-11-02T08:39:16.481 INFO:tasks.workunit.client.0.smithi116.stdout: "pools": [ 2023-11-02T08:39:16.481 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.482 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": 1, 2023-11-02T08:39:16.482 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_name": "test", 2023-11-02T08:39:16.482 INFO:tasks.workunit.client.0.smithi116.stdout: "create_time": "2023-11-02T08:37:06.652604+0000", 2023-11-02T08:39:16.482 INFO:tasks.workunit.client.0.smithi116.stdout: "flags": 1, 2023-11-02T08:39:16.482 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_names": "hashpspool", 2023-11-02T08:39:16.483 INFO:tasks.workunit.client.0.smithi116.stdout: "type": 1, 2023-11-02T08:39:16.483 INFO:tasks.workunit.client.0.smithi116.stdout: "size": 3, 2023-11-02T08:39:16.483 INFO:tasks.workunit.client.0.smithi116.stdout: "min_size": 2, 2023-11-02T08:39:16.483 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_rule": 0, 2023-11-02T08:39:16.483 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_count": 0, 2023-11-02T08:39:16.483 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_target": 0, 2023-11-02T08:39:16.484 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_barrier": 0, 2023-11-02T08:39:16.484 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-11-02T08:39:16.484 INFO:tasks.workunit.client.0.smithi116.stdout: "object_hash": 2, 2023-11-02T08:39:16.484 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_autoscale_mode": "off", 2023-11-02T08:39:16.484 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num": 1, 2023-11-02T08:39:16.485 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_placement_num": 1, 2023-11-02T08:39:16.485 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_placement_num_target": 1, 2023-11-02T08:39:16.485 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num_target": 1, 2023-11-02T08:39:16.485 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num_pending": 1, 2023-11-02T08:39:16.485 INFO:tasks.workunit.client.0.smithi116.stdout: "last_pg_merge_meta": { 2023-11-02T08:39:16.486 INFO:tasks.workunit.client.0.smithi116.stdout: "source_pgid": "0.0", 2023-11-02T08:39:16.486 INFO:tasks.workunit.client.0.smithi116.stdout: "ready_epoch": 0, 2023-11-02T08:39:16.486 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_started": 0, 2023-11-02T08:39:16.486 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_clean": 0, 2023-11-02T08:39:16.486 INFO:tasks.workunit.client.0.smithi116.stdout: "source_version": "0'0", 2023-11-02T08:39:16.487 INFO:tasks.workunit.client.0.smithi116.stdout: "target_version": "0'0" 2023-11-02T08:39:16.487 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.487 INFO:tasks.workunit.client.0.smithi116.stdout: "last_change": "29", 2023-11-02T08:39:16.487 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend": "0", 2023-11-02T08:39:16.487 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend_prenautilus": "0", 2023-11-02T08:39:16.488 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend_preluminous": "0", 2023-11-02T08:39:16.488 INFO:tasks.workunit.client.0.smithi116.stdout: "auid": 0, 2023-11-02T08:39:16.488 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_mode": "selfmanaged", 2023-11-02T08:39:16.488 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_seq": 0, 2023-11-02T08:39:16.488 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_epoch": 0, 2023-11-02T08:39:16.488 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_snaps": [], 2023-11-02T08:39:16.489 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_snaps": "[]", 2023-11-02T08:39:16.489 INFO:tasks.workunit.client.0.smithi116.stdout: "quota_max_bytes": 0, 2023-11-02T08:39:16.489 INFO:tasks.workunit.client.0.smithi116.stdout: "quota_max_objects": 0, 2023-11-02T08:39:16.489 INFO:tasks.workunit.client.0.smithi116.stdout: "tiers": [], 2023-11-02T08:39:16.489 INFO:tasks.workunit.client.0.smithi116.stdout: "tier_of": -1, 2023-11-02T08:39:16.490 INFO:tasks.workunit.client.0.smithi116.stdout: "read_tier": -1, 2023-11-02T08:39:16.490 INFO:tasks.workunit.client.0.smithi116.stdout: "write_tier": -1, 2023-11-02T08:39:16.490 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_mode": "none", 2023-11-02T08:39:16.490 INFO:tasks.workunit.client.0.smithi116.stdout: "target_max_bytes": 0, 2023-11-02T08:39:16.490 INFO:tasks.workunit.client.0.smithi116.stdout: "target_max_objects": 0, 2023-11-02T08:39:16.491 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-11-02T08:39:16.491 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-11-02T08:39:16.491 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_full_ratio_micro": 800000, 2023-11-02T08:39:16.491 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_min_flush_age": 0, 2023-11-02T08:39:16.491 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_min_evict_age": 0, 2023-11-02T08:39:16.492 INFO:tasks.workunit.client.0.smithi116.stdout: "erasure_code_profile": "", 2023-11-02T08:39:16.492 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_params": { 2023-11-02T08:39:16.492 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "none" 2023-11-02T08:39:16.492 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.492 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_period": 0, 2023-11-02T08:39:16.493 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_count": 0, 2023-11-02T08:39:16.493 INFO:tasks.workunit.client.0.smithi116.stdout: "use_gmt_hitset": true, 2023-11-02T08:39:16.493 INFO:tasks.workunit.client.0.smithi116.stdout: "min_read_recency_for_promote": 0, 2023-11-02T08:39:16.493 INFO:tasks.workunit.client.0.smithi116.stdout: "min_write_recency_for_promote": 0, 2023-11-02T08:39:16.493 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_grade_decay_rate": 0, 2023-11-02T08:39:16.493 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_search_last_n": 0, 2023-11-02T08:39:16.494 INFO:tasks.workunit.client.0.smithi116.stdout: "grade_table": [], 2023-11-02T08:39:16.494 INFO:tasks.workunit.client.0.smithi116.stdout: "stripe_width": 0, 2023-11-02T08:39:16.494 INFO:tasks.workunit.client.0.smithi116.stdout: "expected_num_objects": 0, 2023-11-02T08:39:16.494 INFO:tasks.workunit.client.0.smithi116.stdout: "fast_read": false, 2023-11-02T08:39:16.494 INFO:tasks.workunit.client.0.smithi116.stdout: "options": {}, 2023-11-02T08:39:16.495 INFO:tasks.workunit.client.0.smithi116.stdout: "application_metadata": {}, 2023-11-02T08:39:16.495 INFO:tasks.workunit.client.0.smithi116.stdout: "read_balance": { 2023-11-02T08:39:16.495 INFO:tasks.workunit.client.0.smithi116.stdout: "score_acting": 2.9900000095367432, 2023-11-02T08:39:16.495 INFO:tasks.workunit.client.0.smithi116.stdout: "score_stable": 2.9900000095367432, 2023-11-02T08:39:16.495 INFO:tasks.workunit.client.0.smithi116.stdout: "optimal_score": 0.67000001668930054, 2023-11-02T08:39:16.496 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_score_acting": 2, 2023-11-02T08:39:16.496 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_score_stable": 2, 2023-11-02T08:39:16.496 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity_weighted": 1, 2023-11-02T08:39:16.496 INFO:tasks.workunit.client.0.smithi116.stdout: "average_primary_affinity": 1, 2023-11-02T08:39:16.496 INFO:tasks.workunit.client.0.smithi116.stdout: "average_primary_affinity_weighted": 1 2023-11-02T08:39:16.497 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.497 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.497 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:39:16.497 INFO:tasks.workunit.client.0.smithi116.stdout: "osds": [ 2023-11-02T08:39:16.497 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.497 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:39:16.498 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "b4aaa3b4-59f6-4f52-81b9-50100a210eb6", 2023-11-02T08:39:16.498 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 1, 2023-11-02T08:39:16.498 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:39:16.498 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:39:16.499 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:39:16.499 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 5, 2023-11-02T08:39:16.499 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 29, 2023-11-02T08:39:16.499 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 36, 2023-11-02T08:39:16.499 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 41, 2023-11-02T08:39:16.499 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 30, 2023-11-02T08:39:16.500 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:39:16.500 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:39:16.500 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:39:16.500 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.500 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:39:16.501 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6802", 2023-11-02T08:39:16.501 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106338 2023-11-02T08:39:16.501 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.501 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.501 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:39:16.502 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6803", 2023-11-02T08:39:16.502 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106338 2023-11-02T08:39:16.502 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.502 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.502 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.503 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:39:16.503 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:39:16.503 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.503 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:39:16.503 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6804", 2023-11-02T08:39:16.504 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106338 2023-11-02T08:39:16.504 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.504 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.504 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:39:16.504 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6805", 2023-11-02T08:39:16.504 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106338 2023-11-02T08:39:16.505 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.505 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.505 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.505 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:39:16.505 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:39:16.506 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.506 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:39:16.506 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6808", 2023-11-02T08:39:16.506 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106338 2023-11-02T08:39:16.506 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.507 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.507 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:39:16.507 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6809", 2023-11-02T08:39:16.507 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106338 2023-11-02T08:39:16.507 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.508 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.508 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.508 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:39:16.508 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:39:16.508 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.509 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:39:16.509 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6806", 2023-11-02T08:39:16.509 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106338 2023-11-02T08:39:16.509 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.509 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.509 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:39:16.510 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6807", 2023-11-02T08:39:16.510 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106338 2023-11-02T08:39:16.510 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.510 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.510 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.511 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6803/106338", 2023-11-02T08:39:16.511 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6805/106338", 2023-11-02T08:39:16.511 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6809/106338", 2023-11-02T08:39:16.511 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6807/106338", 2023-11-02T08:39:16.511 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:39:16.512 INFO:tasks.workunit.client.0.smithi116.stdout: "exists", 2023-11-02T08:39:16.512 INFO:tasks.workunit.client.0.smithi116.stdout: "up" 2023-11-02T08:39:16.512 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.512 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.512 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.513 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:39:16.513 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "3aac1df4-eb52-4ae6-947d-2e4756e32297", 2023-11-02T08:39:16.513 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 0, 2023-11-02T08:39:16.513 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:39:16.513 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:39:16.513 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:39:16.514 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 0, 2023-11-02T08:39:16.514 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 0, 2023-11-02T08:39:16.514 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 10, 2023-11-02T08:39:16.514 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 23, 2023-11-02T08:39:16.514 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 31, 2023-11-02T08:39:16.515 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:39:16.515 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:39:16.515 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:39:16.515 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.515 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:39:16.516 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6810", 2023-11-02T08:39:16.516 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 100309 2023-11-02T08:39:16.516 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.516 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.516 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:39:16.517 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6811", 2023-11-02T08:39:16.517 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 100309 2023-11-02T08:39:16.517 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.517 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.517 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.517 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:39:16.518 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:39:16.518 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.518 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:39:16.518 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6812", 2023-11-02T08:39:16.518 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 100309 2023-11-02T08:39:16.519 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.519 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.519 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:39:16.519 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6813", 2023-11-02T08:39:16.519 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 100309 2023-11-02T08:39:16.520 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.520 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.520 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.520 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:39:16.520 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:39:16.521 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.521 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:39:16.521 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6816", 2023-11-02T08:39:16.521 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 100309 2023-11-02T08:39:16.521 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.521 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.522 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:39:16.522 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6817", 2023-11-02T08:39:16.522 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 100309 2023-11-02T08:39:16.522 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.522 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.523 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.523 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:39:16.523 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:39:16.523 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.523 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:39:16.524 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6814", 2023-11-02T08:39:16.524 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 100309 2023-11-02T08:39:16.524 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.524 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.525 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:39:16.525 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6815", 2023-11-02T08:39:16.525 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 100309 2023-11-02T08:39:16.525 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.525 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.526 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.526 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6811/100309", 2023-11-02T08:39:16.526 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6813/100309", 2023-11-02T08:39:16.526 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6817/100309", 2023-11-02T08:39:16.526 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6815/100309", 2023-11-02T08:39:16.527 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:39:16.527 INFO:tasks.workunit.client.0.smithi116.stdout: "exists" 2023-11-02T08:39:16.527 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.527 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.527 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.528 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:39:16.528 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "d98f6933-2821-4359-90f4-f9b40e2a8e71", 2023-11-02T08:39:16.528 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 1, 2023-11-02T08:39:16.528 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:39:16.528 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:39:16.529 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:39:16.529 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 15, 2023-11-02T08:39:16.529 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 29, 2023-11-02T08:39:16.529 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 38, 2023-11-02T08:39:16.529 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 0, 2023-11-02T08:39:16.530 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 32, 2023-11-02T08:39:16.530 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:39:16.530 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:39:16.530 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:39:16.531 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.531 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:39:16.531 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6810", 2023-11-02T08:39:16.531 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106908 2023-11-02T08:39:16.531 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.531 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.532 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:39:16.532 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6811", 2023-11-02T08:39:16.532 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106908 2023-11-02T08:39:16.532 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.532 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.533 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.533 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:39:16.533 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:39:16.533 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.533 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:39:16.534 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6812", 2023-11-02T08:39:16.534 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106908 2023-11-02T08:39:16.534 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.534 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.534 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:39:16.535 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6813", 2023-11-02T08:39:16.535 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106908 2023-11-02T08:39:16.535 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.535 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.535 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.536 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:39:16.536 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:39:16.536 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.536 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:39:16.536 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6816", 2023-11-02T08:39:16.536 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106908 2023-11-02T08:39:16.537 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.537 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.537 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:39:16.537 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6817", 2023-11-02T08:39:16.537 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106908 2023-11-02T08:39:16.538 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.538 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.538 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.538 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:39:16.538 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:39:16.539 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.539 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:39:16.539 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6814", 2023-11-02T08:39:16.539 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106908 2023-11-02T08:39:16.539 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.540 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.540 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:39:16.540 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6815", 2023-11-02T08:39:16.540 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 106908 2023-11-02T08:39:16.540 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.541 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.541 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.541 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6811/106908", 2023-11-02T08:39:16.541 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6813/106908", 2023-11-02T08:39:16.541 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6817/106908", 2023-11-02T08:39:16.541 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6815/106908", 2023-11-02T08:39:16.542 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:39:16.542 INFO:tasks.workunit.client.0.smithi116.stdout: "exists", 2023-11-02T08:39:16.542 INFO:tasks.workunit.client.0.smithi116.stdout: "up" 2023-11-02T08:39:16.542 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.542 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.543 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:39:16.543 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_xinfo": [ 2023-11-02T08:39:16.543 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.543 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:39:16.543 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:37:50.470440+0000", 2023-11-02T08:39:16.544 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:39:16.544 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:39:16.544 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:39:16.544 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:39:16.544 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:35:56.521167+0000", 2023-11-02T08:39:16.545 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:39:16.545 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.545 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.545 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:39:16.545 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:37:51.046017+0000", 2023-11-02T08:39:16.546 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:39:16.546 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:39:16.546 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:39:16.546 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:39:16.546 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:36:29.381154+0000", 2023-11-02T08:39:16.547 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:39:16.547 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.547 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.547 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:39:16.547 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:37:51.569072+0000", 2023-11-02T08:39:16.547 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:39:16.548 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:39:16.548 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:39:16.548 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:39:16.548 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:37:02.597739+0000", 2023-11-02T08:39:16.548 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:39:16.549 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.549 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:39:16.549 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap": [], 2023-11-02T08:39:16.549 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap_items": [], 2023-11-02T08:39:16.549 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap_primaries": [], 2023-11-02T08:39:16.550 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_temp": [], 2023-11-02T08:39:16.550 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_temp": [], 2023-11-02T08:39:16.550 INFO:tasks.workunit.client.0.smithi116.stdout: "blocklist": {}, 2023-11-02T08:39:16.550 INFO:tasks.workunit.client.0.smithi116.stdout: "range_blocklist": {}, 2023-11-02T08:39:16.550 INFO:tasks.workunit.client.0.smithi116.stdout: "erasure_code_profiles": { 2023-11-02T08:39:16.551 INFO:tasks.workunit.client.0.smithi116.stdout: "default": { 2023-11-02T08:39:16.551 INFO:tasks.workunit.client.0.smithi116.stdout: "k": "2", 2023-11-02T08:39:16.551 INFO:tasks.workunit.client.0.smithi116.stdout: "m": "2", 2023-11-02T08:39:16.551 INFO:tasks.workunit.client.0.smithi116.stdout: "plugin": "jerasure", 2023-11-02T08:39:16.551 INFO:tasks.workunit.client.0.smithi116.stdout: "technique": "reed_sol_van" 2023-11-02T08:39:16.552 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.552 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.552 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_snaps_queue": [], 2023-11-02T08:39:16.552 INFO:tasks.workunit.client.0.smithi116.stdout: "new_removed_snaps": [], 2023-11-02T08:39:16.552 INFO:tasks.workunit.client.0.smithi116.stdout: "new_purged_snaps": [], 2023-11-02T08:39:16.552 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_node_flags": {}, 2023-11-02T08:39:16.553 INFO:tasks.workunit.client.0.smithi116.stdout: "device_class_flags": {}, 2023-11-02T08:39:16.553 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode": { 2023-11-02T08:39:16.553 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode_enabled": false, 2023-11-02T08:39:16.553 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_bucket_count": 0, 2023-11-02T08:39:16.553 INFO:tasks.workunit.client.0.smithi116.stdout: "degraded_stretch_mode": 0, 2023-11-02T08:39:16.554 INFO:tasks.workunit.client.0.smithi116.stdout: "recovering_stretch_mode": 0, 2023-11-02T08:39:16.554 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode_bucket": 0 2023-11-02T08:39:16.554 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.554 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.554 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_metadata": [ 2023-11-02T08:39:16.555 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.555 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:39:16.555 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:39:16.555 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6804/106338,v1:127.0.0.1:6805/106338]", 2023-11-02T08:39:16.555 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:39:16.556 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:39:16.556 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:39:16.556 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:39:16.556 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:39:16.556 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:39:16.556 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:39:16.557 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:39:16.557 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:39:16.557 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:39:16.557 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2023-11-02T08:39:16.557 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:39:16.558 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:39:16.558 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:39:16.558 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:39:16.558 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:39:16.558 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:39:16.559 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:39:16.559 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:39:16.559 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:39:16.559 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:39:16.559 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:35:31.038465Z", 2023-11-02T08:39:16.560 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:39:16.560 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:39:16.560 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:39:16.560 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:39:16.560 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:39:16.561 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:39:16.561 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:39:16.561 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6802/106338,v1:127.0.0.1:6803/106338]", 2023-11-02T08:39:16.561 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:39:16.561 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/106338,v1:127.0.0.1:6809/106338]", 2023-11-02T08:39:16.561 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/106338,v1:127.0.0.1:6807/106338]", 2023-11-02T08:39:16.562 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:39:16.562 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:39:16.562 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:39:16.562 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:39:16.562 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:39:16.563 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:39:16.563 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:39:16.563 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:39:16.563 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:39:16.563 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/0", 2023-11-02T08:39:16.564 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:39:16.564 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:39:16.564 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:39:16.564 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.564 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.565 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:39:16.565 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:39:16.565 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6812/100309,v1:127.0.0.1:6813/100309]", 2023-11-02T08:39:16.565 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:39:16.565 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:39:16.566 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:39:16.566 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:39:16.566 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:39:16.566 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:39:16.566 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:39:16.566 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:39:16.567 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:39:16.567 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:39:16.567 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2023-11-02T08:39:16.567 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:39:16.567 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:39:16.568 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:39:16.568 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:39:16.568 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:39:16.568 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:39:16.568 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:39:16.569 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:39:16.569 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:39:16.569 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:39:16.569 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:36:02.798675Z", 2023-11-02T08:39:16.569 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:39:16.570 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:39:16.570 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:39:16.570 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:39:16.570 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:39:16.570 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:39:16.571 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:39:16.571 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6810/100309,v1:127.0.0.1:6811/100309]", 2023-11-02T08:39:16.571 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:39:16.571 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/100309,v1:127.0.0.1:6817/100309]", 2023-11-02T08:39:16.571 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/100309,v1:127.0.0.1:6815/100309]", 2023-11-02T08:39:16.571 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:39:16.572 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:39:16.572 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:39:16.572 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:39:16.572 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:39:16.572 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:39:16.573 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:39:16.573 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:39:16.573 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:39:16.573 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/1", 2023-11-02T08:39:16.573 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:39:16.574 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:39:16.574 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:39:16.574 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.574 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.574 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:39:16.575 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:39:16.575 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6812/106908,v1:127.0.0.1:6813/106908]", 2023-11-02T08:39:16.575 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:39:16.575 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:39:16.575 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:39:16.575 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:39:16.576 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:39:16.576 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:39:16.576 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:39:16.576 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:39:16.576 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:39:16.577 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:39:16.577 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2023-11-02T08:39:16.577 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:39:16.577 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:39:16.577 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:39:16.578 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:39:16.578 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:39:16.578 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:39:16.578 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:39:16.578 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:39:16.579 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:39:16.579 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:39:16.579 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:36:36.600371Z", 2023-11-02T08:39:16.579 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:39:16.579 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:39:16.580 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:39:16.580 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:39:16.580 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:39:16.580 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:39:16.580 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:39:16.580 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6810/106908,v1:127.0.0.1:6811/106908]", 2023-11-02T08:39:16.581 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:39:16.581 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/106908,v1:127.0.0.1:6817/106908]", 2023-11-02T08:39:16.581 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/106908,v1:127.0.0.1:6815/106908]", 2023-11-02T08:39:16.581 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:39:16.581 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:39:16.582 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:39:16.582 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:39:16.582 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:39:16.582 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:39:16.582 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:39:16.583 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:39:16.583 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:39:16.583 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/2", 2023-11-02T08:39:16.583 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:39:16.583 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:39:16.584 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:39:16.584 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.584 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:39:16.584 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_clean_epochs": { 2023-11-02T08:39:16.584 INFO:tasks.workunit.client.0.smithi116.stdout: "min_last_epoch_clean": 0, 2023-11-02T08:39:16.584 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_clean": { 2023-11-02T08:39:16.585 INFO:tasks.workunit.client.0.smithi116.stdout: "per_pool": [] 2023-11-02T08:39:16.585 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.585 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_epochs": [ 2023-11-02T08:39:16.585 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.585 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:39:16.586 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 36 2023-11-02T08:39:16.586 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.586 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.586 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:39:16.586 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 10 2023-11-02T08:39:16.587 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.587 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.587 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:39:16.587 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 39 2023-11-02T08:39:16.587 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.588 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.588 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.588 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_first_committed": 1, 2023-11-02T08:39:16.588 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_last_committed": 42, 2023-11-02T08:39:16.588 INFO:tasks.workunit.client.0.smithi116.stdout: "crushmap": { 2023-11-02T08:39:16.589 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": [ 2023-11-02T08:39:16.589 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.589 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:39:16.589 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.0", 2023-11-02T08:39:16.589 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:39:16.589 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.590 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.590 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:39:16.590 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.1", 2023-11-02T08:39:16.590 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:39:16.590 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.591 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.591 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:39:16.591 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.2", 2023-11-02T08:39:16.591 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:39:16.591 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.592 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:39:16.592 INFO:tasks.workunit.client.0.smithi116.stdout: "types": [ 2023-11-02T08:39:16.592 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.592 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 0, 2023-11-02T08:39:16.592 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd" 2023-11-02T08:39:16.593 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.593 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.593 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:39:16.593 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "host" 2023-11-02T08:39:16.593 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.593 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.594 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 2, 2023-11-02T08:39:16.594 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "chassis" 2023-11-02T08:39:16.594 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.594 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.594 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 3, 2023-11-02T08:39:16.595 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "rack" 2023-11-02T08:39:16.595 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.595 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.595 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 4, 2023-11-02T08:39:16.596 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "row" 2023-11-02T08:39:16.596 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.596 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.596 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 5, 2023-11-02T08:39:16.596 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "pdu" 2023-11-02T08:39:16.597 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.597 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.597 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 6, 2023-11-02T08:39:16.597 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "pod" 2023-11-02T08:39:16.597 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.597 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.598 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 7, 2023-11-02T08:39:16.598 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "room" 2023-11-02T08:39:16.598 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.598 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.598 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 8, 2023-11-02T08:39:16.599 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "datacenter" 2023-11-02T08:39:16.599 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.599 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.599 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 9, 2023-11-02T08:39:16.599 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "zone" 2023-11-02T08:39:16.600 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.600 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.600 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 10, 2023-11-02T08:39:16.600 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "region" 2023-11-02T08:39:16.600 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.601 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.601 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:39:16.601 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "root" 2023-11-02T08:39:16.601 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.601 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:39:16.601 INFO:tasks.workunit.client.0.smithi116.stdout: "buckets": [ 2023-11-02T08:39:16.602 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.602 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -1, 2023-11-02T08:39:16.602 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "default", 2023-11-02T08:39:16.602 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:39:16.602 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "root", 2023-11-02T08:39:16.603 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:39:16.603 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:39:16.603 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:39:16.603 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:39:16.603 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.604 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -3, 2023-11-02T08:39:16.604 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:39:16.604 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:39:16.604 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.604 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.605 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.605 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.605 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -2, 2023-11-02T08:39:16.605 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "default~hdd", 2023-11-02T08:39:16.605 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:39:16.605 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "root", 2023-11-02T08:39:16.606 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:39:16.606 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:39:16.606 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:39:16.606 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:39:16.606 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.607 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -4, 2023-11-02T08:39:16.607 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:39:16.607 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:39:16.607 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.607 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.608 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.608 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.608 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -3, 2023-11-02T08:39:16.608 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "smithi116", 2023-11-02T08:39:16.608 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:39:16.609 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "host", 2023-11-02T08:39:16.609 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:39:16.609 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:39:16.609 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:39:16.609 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:39:16.609 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.610 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:39:16.610 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:39:16.610 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:39:16.610 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.610 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.611 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:39:16.611 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:39:16.611 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 1 2023-11-02T08:39:16.611 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.611 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.612 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:39:16.612 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:39:16.612 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 2 2023-11-02T08:39:16.612 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.612 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.613 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.613 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.613 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -4, 2023-11-02T08:39:16.613 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "smithi116~hdd", 2023-11-02T08:39:16.613 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:39:16.613 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "host", 2023-11-02T08:39:16.614 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:39:16.614 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:39:16.614 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:39:16.614 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:39:16.614 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.615 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:39:16.615 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:39:16.615 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:39:16.615 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.615 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.616 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:39:16.616 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:39:16.616 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 1 2023-11-02T08:39:16.616 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.616 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.616 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:39:16.617 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:39:16.617 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 2 2023-11-02T08:39:16.617 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.617 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.617 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.618 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:39:16.618 INFO:tasks.workunit.client.0.smithi116.stdout: "rules": [ 2023-11-02T08:39:16.618 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.618 INFO:tasks.workunit.client.0.smithi116.stdout: "rule_id": 0, 2023-11-02T08:39:16.618 INFO:tasks.workunit.client.0.smithi116.stdout: "rule_name": "replicated_rule", 2023-11-02T08:39:16.619 INFO:tasks.workunit.client.0.smithi116.stdout: "type": 1, 2023-11-02T08:39:16.619 INFO:tasks.workunit.client.0.smithi116.stdout: "steps": [ 2023-11-02T08:39:16.619 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.619 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "take", 2023-11-02T08:39:16.619 INFO:tasks.workunit.client.0.smithi116.stdout: "item": -1, 2023-11-02T08:39:16.620 INFO:tasks.workunit.client.0.smithi116.stdout: "item_name": "default" 2023-11-02T08:39:16.620 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.620 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.620 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "choose_firstn", 2023-11-02T08:39:16.620 INFO:tasks.workunit.client.0.smithi116.stdout: "num": 0, 2023-11-02T08:39:16.620 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "osd" 2023-11-02T08:39:16.621 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.621 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.621 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "emit" 2023-11-02T08:39:16.621 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.621 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:39:16.622 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.622 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:39:16.622 INFO:tasks.workunit.client.0.smithi116.stdout: "tunables": { 2023-11-02T08:39:16.622 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_local_tries": 0, 2023-11-02T08:39:16.622 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_local_fallback_tries": 0, 2023-11-02T08:39:16.623 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_total_tries": 50, 2023-11-02T08:39:16.623 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_descend_once": 1, 2023-11-02T08:39:16.623 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_vary_r": 1, 2023-11-02T08:39:16.623 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_stable": 1, 2023-11-02T08:39:16.623 INFO:tasks.workunit.client.0.smithi116.stdout: "straw_calc_version": 1, 2023-11-02T08:39:16.624 INFO:tasks.workunit.client.0.smithi116.stdout: "allowed_bucket_algs": 54, 2023-11-02T08:39:16.624 INFO:tasks.workunit.client.0.smithi116.stdout: "profile": "jewel", 2023-11-02T08:39:16.624 INFO:tasks.workunit.client.0.smithi116.stdout: "optimal_tunables": 1, 2023-11-02T08:39:16.624 INFO:tasks.workunit.client.0.smithi116.stdout: "legacy_tunables": 0, 2023-11-02T08:39:16.624 INFO:tasks.workunit.client.0.smithi116.stdout: "minimum_required_version": "jewel", 2023-11-02T08:39:16.625 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables": 1, 2023-11-02T08:39:16.625 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables2": 1, 2023-11-02T08:39:16.625 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v2_rules": 0, 2023-11-02T08:39:16.625 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables3": 1, 2023-11-02T08:39:16.625 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v3_rules": 0, 2023-11-02T08:39:16.625 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v4_buckets": 1, 2023-11-02T08:39:16.626 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables5": 1, 2023-11-02T08:39:16.626 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v5_rules": 0 2023-11-02T08:39:16.626 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.626 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_args": {} 2023-11-02T08:39:16.626 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.627 INFO:tasks.workunit.client.0.smithi116.stdout: "fsmap": { 2023-11-02T08:39:16.627 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 1, 2023-11-02T08:39:16.627 INFO:tasks.workunit.client.0.smithi116.stdout: "default_fscid": -1, 2023-11-02T08:39:16.627 INFO:tasks.workunit.client.0.smithi116.stdout: "compat": { 2023-11-02T08:39:16.627 INFO:tasks.workunit.client.0.smithi116.stdout: "compat": {}, 2023-11-02T08:39:16.628 INFO:tasks.workunit.client.0.smithi116.stdout: "ro_compat": {}, 2023-11-02T08:39:16.628 INFO:tasks.workunit.client.0.smithi116.stdout: "incompat": { 2023-11-02T08:39:16.628 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_1": "base v0.20", 2023-11-02T08:39:16.628 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_2": "client writeable ranges", 2023-11-02T08:39:16.628 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_3": "default file layouts on dirs", 2023-11-02T08:39:16.629 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_4": "dir inode in separate object", 2023-11-02T08:39:16.629 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_5": "mds uses versioned encoding", 2023-11-02T08:39:16.629 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_6": "dirfrag is stored in omap", 2023-11-02T08:39:16.629 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_8": "no anchor table", 2023-11-02T08:39:16.630 INFO:tasks.workunit.client.0.smithi116.stderr:report 1299895739 2023-11-02T08:39:16.630 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_9": "file layout v2", 2023-11-02T08:39:16.630 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_10": "snaprealm v2" 2023-11-02T08:39:16.630 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.631 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.631 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_flags": { 2023-11-02T08:39:16.631 INFO:tasks.workunit.client.0.smithi116.stdout: "enable_multiple": true, 2023-11-02T08:39:16.631 INFO:tasks.workunit.client.0.smithi116.stdout: "ever_enabled_multiple": true 2023-11-02T08:39:16.631 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.632 INFO:tasks.workunit.client.0.smithi116.stdout: "standbys": [], 2023-11-02T08:39:16.632 INFO:tasks.workunit.client.0.smithi116.stdout: "filesystems": [] 2023-11-02T08:39:16.632 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.632 INFO:tasks.workunit.client.0.smithi116.stdout: "mdsmap_first_committed": 1, 2023-11-02T08:39:16.632 INFO:tasks.workunit.client.0.smithi116.stdout: "mdsmap_last_committed": 1, 2023-11-02T08:39:16.633 INFO:tasks.workunit.client.0.smithi116.stdout: "auth": { 2023-11-02T08:39:16.633 INFO:tasks.workunit.client.0.smithi116.stdout: "first_committed": 1, 2023-11-02T08:39:16.633 INFO:tasks.workunit.client.0.smithi116.stdout: "last_committed": 5, 2023-11-02T08:39:16.633 INFO:tasks.workunit.client.0.smithi116.stdout: "num_secrets": 3 2023-11-02T08:39:16.633 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.634 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg": 1, 2023-11-02T08:39:16.634 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_active": 1, 2023-11-02T08:39:16.634 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_unknown": 0, 2023-11-02T08:39:16.634 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osd": 3, 2023-11-02T08:39:16.634 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_sum": { 2023-11-02T08:39:16.635 INFO:tasks.workunit.client.0.smithi116.stdout: "stat_sum": { 2023-11-02T08:39:16.635 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes": 47947, 2023-11-02T08:39:16.635 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects": 100, 2023-11-02T08:39:16.635 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_clones": 0, 2023-11-02T08:39:16.635 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_copies": 300, 2023-11-02T08:39:16.635 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing_on_primary": 0, 2023-11-02T08:39:16.636 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing": 0, 2023-11-02T08:39:16.636 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_degraded": 100, 2023-11-02T08:39:16.636 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_misplaced": 0, 2023-11-02T08:39:16.636 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_unfound": 0, 2023-11-02T08:39:16.636 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_dirty": 100, 2023-11-02T08:39:16.637 INFO:tasks.workunit.client.0.smithi116.stdout: "num_whiteouts": 0, 2023-11-02T08:39:16.637 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read": 0, 2023-11-02T08:39:16.637 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read_kb": 0, 2023-11-02T08:39:16.637 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write": 101, 2023-11-02T08:39:16.637 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write_kb": 101, 2023-11-02T08:39:16.638 INFO:tasks.workunit.client.0.smithi116.stdout: "num_scrub_errors": 0, 2023-11-02T08:39:16.638 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shallow_scrub_errors": 0, 2023-11-02T08:39:16.638 INFO:tasks.workunit.client.0.smithi116.stdout: "num_deep_scrub_errors": 0, 2023-11-02T08:39:16.638 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_recovered": 0, 2023-11-02T08:39:16.638 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_recovered": 0, 2023-11-02T08:39:16.639 INFO:tasks.workunit.client.0.smithi116.stdout: "num_keys_recovered": 0, 2023-11-02T08:39:16.639 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_omap": 0, 2023-11-02T08:39:16.639 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_hit_set_archive": 0, 2023-11-02T08:39:16.639 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_hit_set_archive": 0, 2023-11-02T08:39:16.639 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush": 0, 2023-11-02T08:39:16.640 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_kb": 0, 2023-11-02T08:39:16.640 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict": 0, 2023-11-02T08:39:16.640 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_kb": 0, 2023-11-02T08:39:16.640 INFO:tasks.workunit.client.0.smithi116.stdout: "num_promote": 0, 2023-11-02T08:39:16.640 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_high": 0, 2023-11-02T08:39:16.640 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_low": 0, 2023-11-02T08:39:16.641 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_some": 0, 2023-11-02T08:39:16.641 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_full": 0, 2023-11-02T08:39:16.641 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_pinned": 0, 2023-11-02T08:39:16.641 INFO:tasks.workunit.client.0.smithi116.stdout: "num_legacy_snapsets": 0, 2023-11-02T08:39:16.641 INFO:tasks.workunit.client.0.smithi116.stdout: "num_large_omap_objects": 0, 2023-11-02T08:39:16.642 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_manifest": 0, 2023-11-02T08:39:16.642 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_bytes": 0, 2023-11-02T08:39:16.642 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_keys": 0, 2023-11-02T08:39:16.642 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_repaired": 0 2023-11-02T08:39:16.642 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.643 INFO:tasks.workunit.client.0.smithi116.stdout: "store_stats": { 2023-11-02T08:39:16.643 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 0, 2023-11-02T08:39:16.643 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 0, 2023-11-02T08:39:16.643 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:39:16.643 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 0, 2023-11-02T08:39:16.643 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 0, 2023-11-02T08:39:16.644 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:39:16.644 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:39:16.644 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:39:16.644 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 0, 2023-11-02T08:39:16.644 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 0 2023-11-02T08:39:16.645 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.645 INFO:tasks.workunit.client.0.smithi116.stdout: "log_size": 17, 2023-11-02T08:39:16.645 INFO:tasks.workunit.client.0.smithi116.stdout: "ondisk_log_size": 17, 2023-11-02T08:39:16.645 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 2, 2023-11-02T08:39:16.645 INFO:tasks.workunit.client.0.smithi116.stdout: "acting": 2, 2023-11-02T08:39:16.646 INFO:tasks.workunit.client.0.smithi116.stdout: "num_store_stats": 0 2023-11-02T08:39:16.646 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.646 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_sum": { 2023-11-02T08:39:16.646 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 0, 2023-11-02T08:39:16.646 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 0, 2023-11-02T08:39:16.647 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pgs": 3, 2023-11-02T08:39:16.647 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osds": 3, 2023-11-02T08:39:16.647 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_osds": 3, 2023-11-02T08:39:16.647 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_omap_osds": 3, 2023-11-02T08:39:16.647 INFO:tasks.workunit.client.0.smithi116.stdout: "kb": 314572800, 2023-11-02T08:39:16.647 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used": 83304, 2023-11-02T08:39:16.648 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_data": 2120, 2023-11-02T08:39:16.648 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_omap": 2, 2023-11-02T08:39:16.648 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_meta": 81149, 2023-11-02T08:39:16.648 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_avail": 314489496, 2023-11-02T08:39:16.648 INFO:tasks.workunit.client.0.smithi116.stdout: "statfs": { 2023-11-02T08:39:16.649 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 322122547200, 2023-11-02T08:39:16.649 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 322037243904, 2023-11-02T08:39:16.649 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:39:16.649 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 2170880, 2023-11-02T08:39:16.649 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 423287, 2023-11-02T08:39:16.650 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:39:16.650 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:39:16.650 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:39:16.650 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 2366, 2023-11-02T08:39:16.650 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 83097282 2023-11-02T08:39:16.650 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.651 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_peers": [], 2023-11-02T08:39:16.651 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_trim_queue_len": 0, 2023-11-02T08:39:16.651 INFO:tasks.workunit.client.0.smithi116.stdout: "num_snap_trimming": 0, 2023-11-02T08:39:16.651 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shards_repaired": 0, 2023-11-02T08:39:16.651 INFO:tasks.workunit.client.0.smithi116.stdout: "op_queue_age_hist": { 2023-11-02T08:39:16.652 INFO:tasks.workunit.client.0.smithi116.stdout: "histogram": [], 2023-11-02T08:39:16.652 INFO:tasks.workunit.client.0.smithi116.stdout: "upper_bound": 1 2023-11-02T08:39:16.652 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.652 INFO:tasks.workunit.client.0.smithi116.stdout: "perf_stat": { 2023-11-02T08:39:16.652 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ms": 0, 2023-11-02T08:39:16.653 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ms": 0, 2023-11-02T08:39:16.653 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ns": 0, 2023-11-02T08:39:16.653 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ns": 0 2023-11-02T08:39:16.653 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.653 INFO:tasks.workunit.client.0.smithi116.stdout: "alerts": [], 2023-11-02T08:39:16.654 INFO:tasks.workunit.client.0.smithi116.stdout: "network_ping_times": [] 2023-11-02T08:39:16.654 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.654 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_sum_by_class": { 2023-11-02T08:39:16.654 INFO:tasks.workunit.client.0.smithi116.stdout: "hdd": { 2023-11-02T08:39:16.654 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 0, 2023-11-02T08:39:16.655 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 0, 2023-11-02T08:39:16.655 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pgs": 3, 2023-11-02T08:39:16.655 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osds": 3, 2023-11-02T08:39:16.655 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_osds": 3, 2023-11-02T08:39:16.655 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_omap_osds": 3, 2023-11-02T08:39:16.655 INFO:tasks.workunit.client.0.smithi116.stdout: "kb": 314572800, 2023-11-02T08:39:16.656 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used": 83304, 2023-11-02T08:39:16.656 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_data": 2120, 2023-11-02T08:39:16.656 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_omap": 2, 2023-11-02T08:39:16.656 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_meta": 81149, 2023-11-02T08:39:16.656 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_avail": 314489496, 2023-11-02T08:39:16.657 INFO:tasks.workunit.client.0.smithi116.stdout: "statfs": { 2023-11-02T08:39:16.657 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 322122547200, 2023-11-02T08:39:16.657 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 322037243904, 2023-11-02T08:39:16.657 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:39:16.657 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 2170880, 2023-11-02T08:39:16.658 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 423287, 2023-11-02T08:39:16.658 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:39:16.658 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:39:16.658 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:39:16.658 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 2366, 2023-11-02T08:39:16.659 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 83097282 2023-11-02T08:39:16.659 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.659 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_peers": [], 2023-11-02T08:39:16.659 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_trim_queue_len": 0, 2023-11-02T08:39:16.659 INFO:tasks.workunit.client.0.smithi116.stdout: "num_snap_trimming": 0, 2023-11-02T08:39:16.659 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shards_repaired": 0, 2023-11-02T08:39:16.660 INFO:tasks.workunit.client.0.smithi116.stdout: "op_queue_age_hist": { 2023-11-02T08:39:16.660 INFO:tasks.workunit.client.0.smithi116.stdout: "histogram": [], 2023-11-02T08:39:16.660 INFO:tasks.workunit.client.0.smithi116.stdout: "upper_bound": 1 2023-11-02T08:39:16.660 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.660 INFO:tasks.workunit.client.0.smithi116.stdout: "perf_stat": { 2023-11-02T08:39:16.661 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ms": 0, 2023-11-02T08:39:16.661 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ms": 0, 2023-11-02T08:39:16.661 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ns": 0, 2023-11-02T08:39:16.661 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ns": 0 2023-11-02T08:39:16.661 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.662 INFO:tasks.workunit.client.0.smithi116.stdout: "alerts": [], 2023-11-02T08:39:16.662 INFO:tasks.workunit.client.0.smithi116.stdout: "network_ping_times": [] 2023-11-02T08:39:16.662 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.662 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.662 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_stats": [ 2023-11-02T08:39:16.665 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.665 INFO:tasks.workunit.client.0.smithi116.stdout: "poolid": 1, 2023-11-02T08:39:16.665 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg": 1, 2023-11-02T08:39:16.666 INFO:tasks.workunit.client.0.smithi116.stdout: "stat_sum": { 2023-11-02T08:39:16.666 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes": 47947, 2023-11-02T08:39:16.666 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects": 100, 2023-11-02T08:39:16.666 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_clones": 0, 2023-11-02T08:39:16.666 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_copies": 300, 2023-11-02T08:39:16.667 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing_on_primary": 0, 2023-11-02T08:39:16.667 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing": 0, 2023-11-02T08:39:16.667 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_degraded": 100, 2023-11-02T08:39:16.667 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_misplaced": 0, 2023-11-02T08:39:16.667 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_unfound": 0, 2023-11-02T08:39:16.668 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_dirty": 100, 2023-11-02T08:39:16.668 INFO:tasks.workunit.client.0.smithi116.stdout: "num_whiteouts": 0, 2023-11-02T08:39:16.668 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read": 0, 2023-11-02T08:39:16.668 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read_kb": 0, 2023-11-02T08:39:16.668 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write": 101, 2023-11-02T08:39:16.669 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write_kb": 101, 2023-11-02T08:39:16.669 INFO:tasks.workunit.client.0.smithi116.stdout: "num_scrub_errors": 0, 2023-11-02T08:39:16.669 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shallow_scrub_errors": 0, 2023-11-02T08:39:16.669 INFO:tasks.workunit.client.0.smithi116.stdout: "num_deep_scrub_errors": 0, 2023-11-02T08:39:16.669 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_recovered": 0, 2023-11-02T08:39:16.669 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_recovered": 0, 2023-11-02T08:39:16.670 INFO:tasks.workunit.client.0.smithi116.stdout: "num_keys_recovered": 0, 2023-11-02T08:39:16.670 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_omap": 0, 2023-11-02T08:39:16.670 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_hit_set_archive": 0, 2023-11-02T08:39:16.670 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_hit_set_archive": 0, 2023-11-02T08:39:16.670 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush": 0, 2023-11-02T08:39:16.671 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_kb": 0, 2023-11-02T08:39:16.671 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict": 0, 2023-11-02T08:39:16.671 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_kb": 0, 2023-11-02T08:39:16.671 INFO:tasks.workunit.client.0.smithi116.stdout: "num_promote": 0, 2023-11-02T08:39:16.671 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_high": 0, 2023-11-02T08:39:16.672 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_low": 0, 2023-11-02T08:39:16.672 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_some": 0, 2023-11-02T08:39:16.672 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_full": 0, 2023-11-02T08:39:16.672 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_pinned": 0, 2023-11-02T08:39:16.672 INFO:tasks.workunit.client.0.smithi116.stdout: "num_legacy_snapsets": 0, 2023-11-02T08:39:16.673 INFO:tasks.workunit.client.0.smithi116.stdout: "num_large_omap_objects": 0, 2023-11-02T08:39:16.673 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_manifest": 0, 2023-11-02T08:39:16.673 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_bytes": 0, 2023-11-02T08:39:16.673 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_keys": 0, 2023-11-02T08:39:16.673 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_repaired": 0 2023-11-02T08:39:16.674 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.674 INFO:tasks.workunit.client.0.smithi116.stdout: "store_stats": { 2023-11-02T08:39:16.674 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 0, 2023-11-02T08:39:16.674 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 0, 2023-11-02T08:39:16.674 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:39:16.674 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 1220608, 2023-11-02T08:39:16.675 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 142522, 2023-11-02T08:39:16.675 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:39:16.675 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:39:16.675 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:39:16.675 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 0, 2023-11-02T08:39:16.676 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 0 2023-11-02T08:39:16.676 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.676 INFO:tasks.workunit.client.0.smithi116.stdout: "log_size": 17, 2023-11-02T08:39:16.676 INFO:tasks.workunit.client.0.smithi116.stdout: "ondisk_log_size": 17, 2023-11-02T08:39:16.676 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 2, 2023-11-02T08:39:16.677 INFO:tasks.workunit.client.0.smithi116.stdout: "acting": 2, 2023-11-02T08:39:16.677 INFO:tasks.workunit.client.0.smithi116.stdout: "num_store_stats": 3 2023-11-02T08:39:16.677 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.677 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:39:16.677 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_stats": [ 2023-11-02T08:39:16.677 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.678 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:39:16.678 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 154618822676 2023-11-02T08:39:16.678 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.678 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.678 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:39:16.679 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 42949672986 2023-11-02T08:39:16.679 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.679 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.679 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:39:16.679 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 163208757267 2023-11-02T08:39:16.680 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.680 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:39:16.680 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_by_state": [ 2023-11-02T08:39:16.680 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.680 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "active+undersized+degraded", 2023-11-02T08:39:16.681 INFO:tasks.workunit.client.0.smithi116.stdout: "num": 1 2023-11-02T08:39:16.681 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.681 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:39:16.681 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_by_osd": [ 2023-11-02T08:39:16.681 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.681 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:39:16.682 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 0, 2023-11-02T08:39:16.682 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 1, 2023-11-02T08:39:16.682 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:39:16.682 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.682 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.683 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:39:16.683 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 1, 2023-11-02T08:39:16.683 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 1, 2023-11-02T08:39:16.683 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:39:16.683 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.684 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.684 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:39:16.684 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 0, 2023-11-02T08:39:16.684 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 0, 2023-11-02T08:39:16.684 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:39:16.685 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.685 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:39:16.685 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snaps": [ 2023-11-02T08:39:16.685 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:39:16.685 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": 1, 2023-11-02T08:39:16.685 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snaps": {} 2023-11-02T08:39:16.686 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.686 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:39:16.686 INFO:tasks.workunit.client.0.smithi116.stdout: "servicemap": { 2023-11-02T08:39:16.686 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 6, 2023-11-02T08:39:16.686 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:39:01.744442+0000", 2023-11-02T08:39:16.687 INFO:tasks.workunit.client.0.smithi116.stdout: "services": {} 2023-11-02T08:39:16.687 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:39:16.687 INFO:tasks.workunit.client.0.smithi116.stdout: "mgrstat_first_committed": 1, 2023-11-02T08:39:16.687 INFO:tasks.workunit.client.0.smithi116.stdout: "mgrstat_last_committed": 105, 2023-11-02T08:39:16.687 INFO:tasks.workunit.client.0.smithi116.stdout: "logm_first_committed": 1, 2023-11-02T08:39:16.688 INFO:tasks.workunit.client.0.smithi116.stdout: "logm_last_committed": 436, 2023-11-02T08:39:16.688 INFO:tasks.workunit.client.0.smithi116.stdout: "paxos": { 2023-11-02T08:39:16.688 INFO:tasks.workunit.client.0.smithi116.stdout: "first_committed": 1, 2023-11-02T08:39:16.688 INFO:tasks.workunit.client.0.smithi116.stdout: "last_committed": 632, 2023-11-02T08:39:16.688 INFO:tasks.workunit.client.0.smithi116.stdout: "last_pn": 0, 2023-11-02T08:39:16.688 INFO:tasks.workunit.client.0.smithi116.stdout: "accepted_pn": 0 2023-11-02T08:39:16.689 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:39:16.689 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T08:39:16.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2023-11-02T08:39:16.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2023-11-02T08:39:16.690 INFO:tasks.workunit.client.0.smithi116.stdout:delay recovery 2023-11-02T08:39:16.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2023-11-02T08:39:16.850 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:39:16.850 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 100 0 0 47947 0 0 17 84 17 active+undersized+degraded 2023-11-02T08:38:03.562008+0000 42'101 42:125 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:37:06.739239+0000 0'0 2023-11-02T08:37:06.739239+0000 0 0 periodic scrub scheduled @ 2023-11-03T16:37:11.730509+0000 0 0 2023-11-02T08:39:16.850 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:39:16.851 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:39:16.851 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:39:16.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2023-11-02T08:39:16.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2023-11-02T08:39:16.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-11-02T08:39:16.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-11-02T08:39:16.862 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:39:16.863 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:39:16.863 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:39:16.863 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.0.asok 2023-11-02T08:39:16.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2023-11-02T08:39:16.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.0.asok set_recovery_delay 100000 2023-11-02T08:39:16.941 INFO:tasks.workunit.client.0.smithi116.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2023-11-02T08:39:16.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2023-11-02T08:39:16.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2023-11-02T08:39:16.950 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-11-02T08:39:16.950 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-11-02T08:39:16.950 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:39:16.950 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:39:16.950 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:39:16.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.2.asok 2023-11-02T08:39:16.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2023-11-02T08:39:16.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.2.asok set_recovery_delay 100000 2023-11-02T08:39:17.030 INFO:tasks.workunit.client.0.smithi116.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2023-11-02T08:39:17.037 INFO:tasks.workunit.client.0.smithi116.stdout:revive divergent 1 2023-11-02T08:39:17.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2023-11-02T08:39:17.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2023-11-02T08:39:17.395 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:39:17.395 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 100 0 0 47947 0 0 17 84 17 active+undersized+degraded 2023-11-02T08:38:03.562008+0000 42'101 42:125 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:37:06.739239+0000 0'0 2023-11-02T08:37:06.739239+0000 0 0 periodic scrub scheduled @ 2023-11-03T16:37:11.730509+0000 0 0 2023-11-02T08:39:17.395 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:39:17.396 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:39:17.396 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:39:17.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2023-11-02T08:39:18.038 INFO:tasks.workunit.client.0.smithi116.stderr:noup is set 2023-11-02T08:39:18.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2023-11-02T08:39:18.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2023-11-02T08:39:18.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:39:18.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2023-11-02T08:39:18.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:39:18.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2023-11-02T08:39:18.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:39:18.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:39:18.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:39:18.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:39:18.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2023-11-02T08:39:18.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2023-11-02T08:39:18.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:39:18.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:39:18.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:39:18.058 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:39:18.058 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:39:18.059 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:39:18.059 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:39:18.059 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:39:18.059 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:39:18.060 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:39:18.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:39:18.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:39:18.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:39:18.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:39:18.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:39:18.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:39:18.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:39:18.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:39:18.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:39:18.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:39:18.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2023-11-02T08:39:18.064 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T08:39:18.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2023-11-02T08:39:18.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:39:18.066 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2023-11-02T08:39:18.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2023-11-02T08:39:18.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:39:18.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:39:18.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:39:18.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:39:18.080+0000 7f093e412500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:39:18.086 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:39:18.082+0000 7f093e412500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:39:18.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:39:18.083+0000 7f093e412500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:39:18.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2023-11-02T08:39:18.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2023-11-02T08:39:18.603 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:39:18.599+0000 7f093e412500 -1 Falling back to public interface 2023-11-02T08:39:21.230 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:39:21.226+0000 7f093e412500 -1 osd.1 29 log_to_monitors true 2023-11-02T08:39:23.520 INFO:tasks.workunit.client.0.smithi116.stdout:delay recovery divergent 2023-11-02T08:39:23.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2023-11-02T08:39:23.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2023-11-02T08:39:23.888 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:39:23.888 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 100 0 0 47947 0 0 17 84 17 active+undersized+degraded 2023-11-02T08:38:03.562008+0000 42'101 43:127 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:37:06.739239+0000 0'0 2023-11-02T08:37:06.739239+0000 0 0 periodic scrub scheduled @ 2023-11-03T16:37:11.730509+0000 0 0 2023-11-02T08:39:23.889 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:39:23.889 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:39:23.889 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:39:23.900 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2023-11-02T08:39:23.900 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-11-02T08:39:23.900 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-11-02T08:39:23.901 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:39:23.901 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:39:23.902 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:39:23.902 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.1.asok 2023-11-02T08:39:23.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2023-11-02T08:39:23.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.1.asok set_recovery_delay 100000 2023-11-02T08:39:23.979 INFO:tasks.workunit.client.0.smithi116.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2023-11-02T08:39:23.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2023-11-02T08:39:24.801 INFO:tasks.workunit.client.0.smithi116.stderr:noup is unset 2023-11-02T08:39:24.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2023-11-02T08:39:24.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:39:24.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:39:24.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:39:24.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:39:24.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:39:24.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:39:24.814 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:39:24.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:39:24.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:39:25.239 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 36 up_thru 41 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/106338,v1:127.0.0.1:6803/106338] [v2:127.0.0.1:6804/106338,v1:127.0.0.1:6805/106338] exists,up b4aaa3b4-59f6-4f52-81b9-50100a210eb6 2023-11-02T08:39:25.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:39:25.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:39:25.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:39:25.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2023-11-02T08:39:25.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:39:25.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:39:25.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:39:25.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:39:25.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:39:25.242 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:39:25.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:39:25.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:39:25.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:39:25.620 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/109436,v1:127.0.0.1:6819/109436] [v2:127.0.0.1:6820/109436,v1:127.0.0.1:6821/109436] exists,up 3aac1df4-eb52-4ae6-947d-2e4756e32297 2023-11-02T08:39:25.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:39:25.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:39:25.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:39:25.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2023-11-02T08:39:25.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:39:25.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T08:39:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:39:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:39:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:39:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:39:25.624 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:39:25.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:39:25.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:39:26.008 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/106908,v1:127.0.0.1:6811/106908] [v2:127.0.0.1:6812/106908,v1:127.0.0.1:6813/106908] exists,up d98f6933-2821-4359-90f4-f9b40e2a8e71 2023-11-02T08:39:26.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:39:26.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:39:26.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:39:26.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2023-11-02T08:39:26.362 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:39:26.362 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 100 0 0 47947 0 0 17 84 17 active+undersized+degraded 2023-11-02T08:38:03.562008+0000 42'101 43:127 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:37:06.739239+0000 0'0 2023-11-02T08:37:06.739239+0000 0 0 periodic scrub scheduled @ 2023-11-03T16:37:11.730509+0000 0 0 2023-11-02T08:39:26.362 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:39:26.363 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:39:26.363 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:39:26.372 INFO:tasks.workunit.client.0.smithi116.stdout:wait for peering 2023-11-02T08:39:26.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2023-11-02T08:39:26.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2023-11-02T08:39:26.730 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:39:26.730 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 100 0 0 47947 0 0 17 84 17 active+undersized+degraded 2023-11-02T08:38:03.562008+0000 42'101 43:127 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:37:06.739239+0000 0'0 2023-11-02T08:37:06.739239+0000 0 0 periodic scrub scheduled @ 2023-11-03T16:37:11.730509+0000 0 0 2023-11-02T08:39:26.730 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:39:26.731 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:39:26.731 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:39:26.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2023-11-02T08:39:26.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2023-11-02T08:39:26.884 INFO:tasks.workunit.client.0.smithi116.stdout:killing divergent 1 2023-11-02T08:39:26.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2023-11-02T08:39:27.264 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:39:27.264 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 100 0 0 47947 0 0 17 84 17 active+undersized+degraded 2023-11-02T08:38:03.562008+0000 42'101 43:127 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:37:06.739239+0000 0'0 2023-11-02T08:37:06.739239+0000 0 0 periodic scrub scheduled @ 2023-11-03T16:37:11.730509+0000 0 0 2023-11-02T08:39:27.265 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:39:27.265 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:39:27.265 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:39:27.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2023-11-02T08:39:27.276 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:39:27.276 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:39:27.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:39:27.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:39:27.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:39:27.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:39:27.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.85082.out 2023-11-02T08:39:27.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.85082.out 2023-11-02T08:39:27.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2023-11-02T08:39:27.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2023-11-02T08:39:27.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2023-11-02T08:39:27.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2023-11-02T08:39:27.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2023-11-02T08:39:27.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.85082.out 2023-11-02T08:39:28.479 INFO:tasks.workunit.client.0.smithi116.stderr:Exporting 1.0 info 1.0( v 48'102 lc 0'0 (29'90,48'102] local-lis/les=47/48 n=101 ec=23/23 lis/c=47/23 les/c/f=48/24/0 sis=47) 2023-11-02T08:39:28.480 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:023dd3d2:::existing_12:head# 2023-11-02T08:39:28.488 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:04c40a0c:::existing_93:head# 2023-11-02T08:39:28.489 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:064ff5fb:::existing_36:head# 2023-11-02T08:39:28.490 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:091f9f1d:::existing_94:head# 2023-11-02T08:39:28.490 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:0a865a93:::existing_54:head# 2023-11-02T08:39:28.492 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:0fed513c:::existing_41:head# 2023-11-02T08:39:28.492 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:13b9e0b8:::existing_92:head# 2023-11-02T08:39:28.492 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:169157fa:::existing_81:head# 2023-11-02T08:39:28.493 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:17731382:::existing_58:head# 2023-11-02T08:39:28.493 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:18a45a39:::existing_43:head# 2023-11-02T08:39:28.493 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:1c160f01:::existing_66:head# 2023-11-02T08:39:28.493 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:1dd6c249:::existing_97:head# 2023-11-02T08:39:28.493 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:227e0c77:::existing_7:head# 2023-11-02T08:39:28.495 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:23e1b063:::existing_59:head# 2023-11-02T08:39:28.497 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:28566edb:::existing_78:head# 2023-11-02T08:39:28.497 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:2a101182:::existing_20:head# 2023-11-02T08:39:28.500 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:2d1519da:::existing_96:head# 2023-11-02T08:39:28.501 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:3078b87a:::existing_86:head# 2023-11-02T08:39:28.501 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:30bde01b:::existing_15:head# 2023-11-02T08:39:28.501 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:317a7e19:::existing_56:head# 2023-11-02T08:39:28.501 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:38b9c7fa:::existing_33:head# 2023-11-02T08:39:28.501 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:3e16d84b:::existing_48:head# 2023-11-02T08:39:28.502 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:45fe8203:::existing_75:head# 2023-11-02T08:39:28.502 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:4e82c9ba:::existing_50:head# 2023-11-02T08:39:28.502 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:5298dcdb:::existing_57:head# 2023-11-02T08:39:28.502 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:52ebaa44:::existing_32:head# 2023-11-02T08:39:28.503 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:53588912:::existing_9:head# 2023-11-02T08:39:28.503 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:5b7e2de6:::existing_84:head# 2023-11-02T08:39:28.503 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:5da0255c:::existing_13:head# 2023-11-02T08:39:28.503 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:5da2dab8:::existing_53:head# 2023-11-02T08:39:28.503 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:5ee3514b:::existing_30:head# 2023-11-02T08:39:28.503 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:602f83fe:::foo:head# 2023-11-02T08:39:28.505 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:60426cc7:::existing_62:head# 2023-11-02T08:39:28.505 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:628918ca:::existing_79:head# 2023-11-02T08:39:28.506 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:63f8ab2d:::existing_77:head# 2023-11-02T08:39:28.506 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:67e1780f:::existing_27:head# 2023-11-02T08:39:28.506 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:68143ca1:::existing_71:head# 2023-11-02T08:39:28.506 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:6bac9509:::existing_29:head# 2023-11-02T08:39:28.507 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:6bbec798:::existing_45:head# 2023-11-02T08:39:28.507 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:6cb23a40:::existing_65:head# 2023-11-02T08:39:28.507 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:7055ec5e:::existing_83:head# 2023-11-02T08:39:28.508 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:70a54e2b:::existing_87:head# 2023-11-02T08:39:28.508 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:750ace21:::existing_19:head# 2023-11-02T08:39:28.508 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:78623b1d:::existing_6:head# 2023-11-02T08:39:28.512 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:79881152:::existing_63:head# 2023-11-02T08:39:28.514 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:7a80c0d5:::existing_99:head# 2023-11-02T08:39:28.515 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:81490677:::existing_76:head# 2023-11-02T08:39:28.515 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:85ec8e5d:::existing_85:head# 2023-11-02T08:39:28.515 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:8e2abbf0:::existing_49:head# 2023-11-02T08:39:28.516 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:8ee0fe0f:::existing_52:head# 2023-11-02T08:39:28.516 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:923c5d4d:::existing_5:head# 2023-11-02T08:39:28.524 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:92731e71:::existing_89:head# 2023-11-02T08:39:28.525 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:928205cf:::existing_82:head# 2023-11-02T08:39:28.525 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:93be2267:::existing_24:head# 2023-11-02T08:39:28.525 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:99ceda2c:::existing_44:head# 2023-11-02T08:39:28.525 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:9dd917ee:::existing_26:head# 2023-11-02T08:39:28.525 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:a28805a6:::existing_72:head# 2023-11-02T08:39:28.526 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:a294a8f6:::existing_16:head# 2023-11-02T08:39:28.526 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:a2f5473a:::existing_42:head# 2023-11-02T08:39:28.526 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:a504a533:::existing_90:head# 2023-11-02T08:39:28.526 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:a691337e:::existing_61:head# 2023-11-02T08:39:28.526 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:a6e8d7ac:::existing_28:head# 2023-11-02T08:39:28.527 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:a9fde86e:::existing_18:head# 2023-11-02T08:39:28.527 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:aa309faa:::existing_68:head# 2023-11-02T08:39:28.527 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:ae40b0ed:::existing_31:head# 2023-11-02T08:39:28.527 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:b279b1fe:::existing_10:head# 2023-11-02T08:39:28.528 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:b343e277:::existing_88:head# 2023-11-02T08:39:28.528 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:b7130e90:::existing_80:head# 2023-11-02T08:39:28.528 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:bc6f7309:::existing_70:head# 2023-11-02T08:39:28.529 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:be8489b7:::existing_38:head# 2023-11-02T08:39:28.529 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:beb4e5e3:::existing_47:head# 2023-11-02T08:39:28.529 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:bebec877:::existing_74:head# 2023-11-02T08:39:28.530 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:c04347d8:::existing_60:head# 2023-11-02T08:39:28.530 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:c156033e:::existing_100:head# 2023-11-02T08:39:28.530 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:c19f5a8e:::existing_8:head# 2023-11-02T08:39:28.530 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:c3e6ebe4:::existing_17:head# 2023-11-02T08:39:28.530 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:c6c5a7c2:::existing_39:head# 2023-11-02T08:39:28.531 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:cc1d0abf:::existing_95:head# 2023-11-02T08:39:28.531 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:ccb3865b:::existing_73:head# 2023-11-02T08:39:28.531 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:cd84e19e:::existing_40:head# 2023-11-02T08:39:28.531 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:d2cd64ce:::existing_37:head# 2023-11-02T08:39:28.531 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:d41a25c9:::existing_22:head# 2023-11-02T08:39:28.532 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:d7f088da:::existing_91:head# 2023-11-02T08:39:28.532 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:d9cbd1d6:::existing_64:head# 2023-11-02T08:39:28.532 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:dcc4ff82:::existing_98:head# 2023-11-02T08:39:28.532 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:dd677bd3:::existing_35:head# 2023-11-02T08:39:28.532 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:dd6d90e7:::existing_46:head# 2023-11-02T08:39:28.533 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:e0e22077:::existing_21:head# 2023-11-02T08:39:28.533 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:e255b710:::existing_55:head# 2023-11-02T08:39:28.533 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:eff1d5db:::existing_14:head# 2023-11-02T08:39:28.533 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:f1b156bf:::existing_23:head# 2023-11-02T08:39:28.533 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:f5ce4199:::existing_34:head# 2023-11-02T08:39:28.533 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:f6809456:::existing_11:head# 2023-11-02T08:39:28.534 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:f699a2a9:::existing_67:head# 2023-11-02T08:39:28.534 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:f9baac43:::existing_25:head# 2023-11-02T08:39:28.534 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:fb76688e:::existing_51:head# 2023-11-02T08:39:28.534 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:fbb92eec:::existing_69:head# 2023-11-02T08:39:28.534 INFO:tasks.workunit.client.0.smithi116.stderr:Export successful 2023-11-02T08:39:28.535 INFO:tasks.workunit.client.0.smithi116.stdout: marking collection for removal 2023-11-02T08:39:28.535 INFO:tasks.workunit.client.0.smithi116.stdout:setting '_remove' omap key 2023-11-02T08:39:28.535 INFO:tasks.workunit.client.0.smithi116.stdout:finish_remove_pgs 1.0_head removing 1.0 2023-11-02T08:39:28.671 INFO:tasks.workunit.client.0.smithi116.stderr:Remove successful 2023-11-02T08:39:29.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.85082.out 2023-11-02T08:39:29.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2023-11-02T08:39:29.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2023-11-02T08:39:29.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2023-11-02T08:39:29.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2023-11-02T08:39:29.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2023-11-02T08:39:29.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.85082.out 2023-11-02T08:39:29.849 INFO:tasks.workunit.client.0.smithi116.stdout:get_pg_num_history pg_num_history pg_num_history(e48 pg_nums {1={23=1}} deleted_pools ) 2023-11-02T08:39:29.849 INFO:tasks.workunit.client.0.smithi116.stdout:Importing pgid 1.0 2023-11-02T08:39:29.850 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:023dd3d2:::existing_12:head# 2023-11-02T08:39:29.850 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:29.941 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:04c40a0c:::existing_93:head# 2023-11-02T08:39:29.941 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:29.974 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:064ff5fb:::existing_36:head# 2023-11-02T08:39:29.975 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.041 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:091f9f1d:::existing_94:head# 2023-11-02T08:39:30.041 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.101 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:0a865a93:::existing_54:head# 2023-11-02T08:39:30.101 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.167 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:0fed513c:::existing_41:head# 2023-11-02T08:39:30.167 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.200 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:13b9e0b8:::existing_92:head# 2023-11-02T08:39:30.201 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.234 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:169157fa:::existing_81:head# 2023-11-02T08:39:30.234 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.267 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:17731382:::existing_58:head# 2023-11-02T08:39:30.267 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.301 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:18a45a39:::existing_43:head# 2023-11-02T08:39:30.301 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.334 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:1c160f01:::existing_66:head# 2023-11-02T08:39:30.334 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.368 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:1dd6c249:::existing_97:head# 2023-11-02T08:39:30.368 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.393 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:227e0c77:::existing_7:head# 2023-11-02T08:39:30.393 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.418 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:23e1b063:::existing_59:head# 2023-11-02T08:39:30.418 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.443 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:28566edb:::existing_78:head# 2023-11-02T08:39:30.443 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.468 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:2a101182:::existing_20:head# 2023-11-02T08:39:30.469 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.493 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:2d1519da:::existing_96:head# 2023-11-02T08:39:30.494 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.518 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:3078b87a:::existing_86:head# 2023-11-02T08:39:30.519 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.543 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:30bde01b:::existing_15:head# 2023-11-02T08:39:30.544 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.569 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:317a7e19:::existing_56:head# 2023-11-02T08:39:30.569 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.594 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:38b9c7fa:::existing_33:head# 2023-11-02T08:39:30.594 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.619 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:3e16d84b:::existing_48:head# 2023-11-02T08:39:30.619 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.644 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:45fe8203:::existing_75:head# 2023-11-02T08:39:30.644 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.669 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:4e82c9ba:::existing_50:head# 2023-11-02T08:39:30.669 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.695 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:5298dcdb:::existing_57:head# 2023-11-02T08:39:30.695 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.719 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:52ebaa44:::existing_32:head# 2023-11-02T08:39:30.720 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.745 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:53588912:::existing_9:head# 2023-11-02T08:39:30.745 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.770 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:5b7e2de6:::existing_84:head# 2023-11-02T08:39:30.770 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.795 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:5da0255c:::existing_13:head# 2023-11-02T08:39:30.796 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.820 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:5da2dab8:::existing_53:head# 2023-11-02T08:39:30.821 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.845 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:5ee3514b:::existing_30:head# 2023-11-02T08:39:30.846 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.871 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:602f83fe:::foo:head# 2023-11-02T08:39:30.871 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.895 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:60426cc7:::existing_62:head# 2023-11-02T08:39:30.896 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.920 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:628918ca:::existing_79:head# 2023-11-02T08:39:30.921 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.946 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:63f8ab2d:::existing_77:head# 2023-11-02T08:39:30.946 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.970 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:67e1780f:::existing_27:head# 2023-11-02T08:39:30.971 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:30.996 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:68143ca1:::existing_71:head# 2023-11-02T08:39:30.996 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.021 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:6bac9509:::existing_29:head# 2023-11-02T08:39:31.021 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.046 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:6bbec798:::existing_45:head# 2023-11-02T08:39:31.046 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.071 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:6cb23a40:::existing_65:head# 2023-11-02T08:39:31.071 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.096 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:7055ec5e:::existing_83:head# 2023-11-02T08:39:31.097 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.121 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:70a54e2b:::existing_87:head# 2023-11-02T08:39:31.122 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.146 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:750ace21:::existing_19:head# 2023-11-02T08:39:31.147 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.172 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:78623b1d:::existing_6:head# 2023-11-02T08:39:31.172 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.197 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:79881152:::existing_63:head# 2023-11-02T08:39:31.197 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.222 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:7a80c0d5:::existing_99:head# 2023-11-02T08:39:31.222 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.247 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:81490677:::existing_76:head# 2023-11-02T08:39:31.247 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.272 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:85ec8e5d:::existing_85:head# 2023-11-02T08:39:31.272 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.297 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:8e2abbf0:::existing_49:head# 2023-11-02T08:39:31.298 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.322 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:8ee0fe0f:::existing_52:head# 2023-11-02T08:39:31.323 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.347 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:923c5d4d:::existing_5:head# 2023-11-02T08:39:31.348 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.373 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:92731e71:::existing_89:head# 2023-11-02T08:39:31.373 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.398 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:928205cf:::existing_82:head# 2023-11-02T08:39:31.398 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.423 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:93be2267:::existing_24:head# 2023-11-02T08:39:31.423 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.448 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:99ceda2c:::existing_44:head# 2023-11-02T08:39:31.448 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.474 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:9dd917ee:::existing_26:head# 2023-11-02T08:39:31.474 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.540 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:a28805a6:::existing_72:head# 2023-11-02T08:39:31.540 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.573 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:a294a8f6:::existing_16:head# 2023-11-02T08:39:31.574 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.608 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:a2f5473a:::existing_42:head# 2023-11-02T08:39:31.609 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.640 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:a504a533:::existing_90:head# 2023-11-02T08:39:31.641 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.674 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:a691337e:::existing_61:head# 2023-11-02T08:39:31.674 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.707 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:a6e8d7ac:::existing_28:head# 2023-11-02T08:39:31.708 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.741 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:a9fde86e:::existing_18:head# 2023-11-02T08:39:31.741 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.774 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:aa309faa:::existing_68:head# 2023-11-02T08:39:31.775 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.808 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:ae40b0ed:::existing_31:head# 2023-11-02T08:39:31.808 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.917 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:b279b1fe:::existing_10:head# 2023-11-02T08:39:31.917 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.950 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:b343e277:::existing_88:head# 2023-11-02T08:39:31.950 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:31.984 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:b7130e90:::existing_80:head# 2023-11-02T08:39:31.984 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.025 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:bc6f7309:::existing_70:head# 2023-11-02T08:39:32.026 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.084 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:be8489b7:::existing_38:head# 2023-11-02T08:39:32.084 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.134 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:beb4e5e3:::existing_47:head# 2023-11-02T08:39:32.134 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.185 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:bebec877:::existing_74:head# 2023-11-02T08:39:32.185 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.218 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:c04347d8:::existing_60:head# 2023-11-02T08:39:32.218 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.252 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:c156033e:::existing_100:head# 2023-11-02T08:39:32.263 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.285 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:c19f5a8e:::existing_8:head# 2023-11-02T08:39:32.285 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.319 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:c3e6ebe4:::existing_17:head# 2023-11-02T08:39:32.319 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.352 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:c6c5a7c2:::existing_39:head# 2023-11-02T08:39:32.353 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.386 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:cc1d0abf:::existing_95:head# 2023-11-02T08:39:32.386 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.410 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:ccb3865b:::existing_73:head# 2023-11-02T08:39:32.411 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.436 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:cd84e19e:::existing_40:head# 2023-11-02T08:39:32.436 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.461 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:d2cd64ce:::existing_37:head# 2023-11-02T08:39:32.461 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.486 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:d41a25c9:::existing_22:head# 2023-11-02T08:39:32.486 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.511 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:d7f088da:::existing_91:head# 2023-11-02T08:39:32.511 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.536 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:d9cbd1d6:::existing_64:head# 2023-11-02T08:39:32.536 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.586 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:dcc4ff82:::existing_98:head# 2023-11-02T08:39:32.587 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.611 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:dd677bd3:::existing_35:head# 2023-11-02T08:39:32.612 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.637 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:dd6d90e7:::existing_46:head# 2023-11-02T08:39:32.637 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.661 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:e0e22077:::existing_21:head# 2023-11-02T08:39:32.662 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.687 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:e255b710:::existing_55:head# 2023-11-02T08:39:32.687 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.712 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:eff1d5db:::existing_14:head# 2023-11-02T08:39:32.712 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.737 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:f1b156bf:::existing_23:head# 2023-11-02T08:39:32.737 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.762 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:f5ce4199:::existing_34:head# 2023-11-02T08:39:32.762 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.788 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:f6809456:::existing_11:head# 2023-11-02T08:39:32.789 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.812 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:f699a2a9:::existing_67:head# 2023-11-02T08:39:32.813 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.837 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:f9baac43:::existing_25:head# 2023-11-02T08:39:32.838 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.863 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:fb76688e:::existing_51:head# 2023-11-02T08:39:32.863 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.889 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:fbb92eec:::existing_69:head# 2023-11-02T08:39:32.889 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:39:32.913 INFO:tasks.workunit.client.0.smithi116.stdout:write_pg epoch 48 info 1.0( v 48'102 lc 0'0 (29'90,48'102] local-lis/les=47/48 n=101 ec=23/23 lis/c=47/23 les/c/f=48/24/0 sis=47) 2023-11-02T08:39:32.938 INFO:tasks.workunit.client.0.smithi116.stdout:Import successful 2023-11-02T08:39:33.454 INFO:tasks.workunit.client.0.smithi116.stdout:reviving divergent 1 2023-11-02T08:39:33.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2023-11-02T08:39:33.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2023-11-02T08:39:33.825 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:39:33.825 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 4 4 0 0 47947 0 0 11 90 11 active+recovering+degraded 2023-11-02T08:39:25.544069+0000 42'101 48:126 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:37:06.739239+0000 0'0 2023-11-02T08:37:06.739239+0000 0 0 periodic scrub scheduled @ 2023-11-03T18:03:38.498073+0000 0 0 2023-11-02T08:39:33.826 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:39:33.826 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:39:33.826 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:39:33.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2023-11-02T08:39:33.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2023-11-02T08:39:33.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:39:33.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2023-11-02T08:39:33.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:39:33.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2023-11-02T08:39:33.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:39:33.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:39:33.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:39:33.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:39:33.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2023-11-02T08:39:33.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2023-11-02T08:39:33.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:39:33.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:39:33.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:39:33.841 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:39:33.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:39:33.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:39:33.842 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:39:33.842 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:39:33.843 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:39:33.843 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:39:33.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:39:33.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:39:33.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:39:33.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:39:33.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:39:33.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:39:33.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:39:33.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:39:33.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:39:33.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:39:33.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2023-11-02T08:39:33.846 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T08:39:33.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2023-11-02T08:39:33.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:39:33.847 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2023-11-02T08:39:33.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2023-11-02T08:39:33.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:39:33.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:39:33.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:39:33.871 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:39:33.868+0000 7fd1c0f6e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:39:33.873 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:39:33.870+0000 7fd1c0f6e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:39:33.874 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:39:33.871+0000 7fd1c0f6e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:39:34.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2023-11-02T08:39:34.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:39:34.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:39:34.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:39:34.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:39:34.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:39:34.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:39:34.223 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:39:34.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:39:34.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:39:34.396 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:39:34.393+0000 7fd1c0f6e500 -1 Falling back to public interface 2023-11-02T08:39:34.650 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/109436,v1:127.0.0.1:6819/109436] [v2:127.0.0.1:6820/109436,v1:127.0.0.1:6821/109436] exists,up 3aac1df4-eb52-4ae6-947d-2e4756e32297 2023-11-02T08:39:34.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:39:34.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:39:34.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:39:34.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2023-11-02T08:39:34.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:39:34.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:39:34.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:39:34.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:39:34.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:39:34.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:39:34.654 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:39:34.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:39:34.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:39:35.026 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/109436,v1:127.0.0.1:6819/109436] [v2:127.0.0.1:6820/109436,v1:127.0.0.1:6821/109436] exists,up 3aac1df4-eb52-4ae6-947d-2e4756e32297 2023-11-02T08:39:35.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:39:35.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:39:35.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:39:35.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2023-11-02T08:39:36.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:39:36.975+0000 7fd1c0f6e500 -1 osd.1 48 log_to_monitors true 2023-11-02T08:39:55.029 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2023-11-02T08:39:55.029 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-11-02T08:39:55.030 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-11-02T08:39:55.030 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:39:55.030 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:39:55.030 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:39:55.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.1.asok 2023-11-02T08:39:55.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2023-11-02T08:39:55.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.1.asok dump_ops_in_flight 2023-11-02T08:39:55.114 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T08:39:55.115 INFO:tasks.workunit.client.0.smithi116.stdout: "ops": [], 2023-11-02T08:39:55.115 INFO:tasks.workunit.client.0.smithi116.stdout: "num_ops": 0 2023-11-02T08:39:55.115 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T08:39:55.122 INFO:tasks.workunit.client.0.smithi116.stdout:allowing recovery 2023-11-02T08:39:55.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2023-11-02T08:39:55.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2023-11-02T08:39:55.480 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:39:55.481 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 101 0 0 0 0 48430 0 0 12 90 12 active+clean 2023-11-02T08:39:38.470447+0000 48'102 52:148 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:37:06.739239+0000 0'0 2023-11-02T08:37:06.739239+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:33:49.447164+0000 0 0 2023-11-02T08:39:55.481 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:39:55.481 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:39:55.482 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:39:55.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2023-11-02T08:39:55.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2023-11-02T08:39:55.599 INFO:tasks.workunit.client.0.smithi116.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2023-11-02T08:39:55.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2023-11-02T08:39:55.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2023-11-02T08:39:55.718 INFO:tasks.workunit.client.0.smithi116.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2023-11-02T08:39:55.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2023-11-02T08:39:55.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2023-11-02T08:39:55.860 INFO:tasks.workunit.client.0.smithi116.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2023-11-02T08:39:55.869 INFO:tasks.workunit.client.0.smithi116.stdout:reading divergent objects 2023-11-02T08:39:55.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2023-11-02T08:39:55.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2023-11-02T08:39:56.228 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:39:56.228 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 101 0 0 0 0 48430 0 0 12 90 12 active+clean 2023-11-02T08:39:38.470447+0000 48'102 52:148 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:37:06.739239+0000 0'0 2023-11-02T08:37:06.739239+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:33:49.447164+0000 0 0 2023-11-02T08:39:56.228 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:39:56.228 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:39:56.229 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:39:56.239 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2023-11-02T08:39:56.239 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2023-11-02T08:39:56.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2023-11-02T08:39:56.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2023-11-02T08:39:56.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2023-11-02T08:39:56.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2023-11-02T08:39:56.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2023-11-02T08:39:56.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2023-11-02T08:39:56.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2023-11-02T08:39:56.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2023-11-02T08:39:56.374 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2023-11-02T08:39:56.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2023-11-02T08:39:56.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2023-11-02T08:39:56.406 INFO:tasks.workunit.client.0.smithi116.stderr:error getting test/newobject_1: (2) No such file or directory 2023-11-02T08:39:56.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2023-11-02T08:39:56.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2023-11-02T08:39:56.439 INFO:tasks.workunit.client.0.smithi116.stderr:error getting test/newobject_2: (2) No such file or directory 2023-11-02T08:39:56.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2023-11-02T08:39:56.442 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2023-11-02T08:39:56.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log 2023-11-02T08:39:56.448 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'106 (29'4) delete 1:70b3bf12:::existing_4:head by client.4635.0:1 2023-11-02T08:37:39.714659+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2023-11-02T08:39:56.449 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 20 _merge_object_divergent_entries: keeping 29'106 (29'4) delete 1:70b3bf12:::existing_4:head by client.4635.0:1 2023-11-02T08:37:39.714659+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2023-11-02T08:39:56.449 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2023-11-02T08:39:56.449 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'106 last_divergent_update: 29'106 2023-11-02T08:39:56.449 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: more recent entry found: 42'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4750.0:1 2023-11-02T08:38:40.169945+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2023-11-02T08:39:56.449 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4622.0:1 2023-11-02T08:37:39.690426+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2023-11-02T08:39:56.450 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 20 _merge_object_divergent_entries: keeping 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4622.0:1 2023-11-02T08:37:39.690426+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2023-11-02T08:39:56.450 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2023-11-02T08:39:56.450 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'101 last_divergent_update: 29'101 2023-11-02T08:39:56.450 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2023-11-02T08:39:56.450 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2023-11-02T08:39:56.451 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'104 (29'3) delete 1:8881e6d8:::existing_3:head by client.4630.0:1 2023-11-02T08:37:39.706508+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2023-11-02T08:39:56.451 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 20 _merge_object_divergent_entries: keeping 29'104 (29'3) delete 1:8881e6d8:::existing_3:head by client.4630.0:1 2023-11-02T08:37:39.706508+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2023-11-02T08:39:56.451 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2023-11-02T08:39:56.451 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'104 last_divergent_update: 29'104 2023-11-02T08:39:56.451 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2023-11-02T08:39:56.452 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2023-11-02T08:39:56.452 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2023-11-02T08:39:56.452 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'104 (29'3) delete 1:8881e6d8:::existing_3:head by client.4630.0:1 2023-11-02T08:37:39.706508+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2023-11-02T08:39:56.452 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2023-11-02T08:39:56.452 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'90 2023-11-02T08:39:56.453 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4623.0:1 2023-11-02T08:37:39.691220+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:39:56.453 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 20 _merge_object_divergent_entries: keeping 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4623.0:1 2023-11-02T08:37:39.691220+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:39:56.453 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2023-11-02T08:39:56.454 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'102 last_divergent_update: 29'102 2023-11-02T08:39:56.454 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2023-11-02T08:39:56.454 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2023-11-02T08:39:56.455 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2023-11-02T08:39:56.455 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4623.0:1 2023-11-02T08:37:39.691220+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:39:56.455 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2023-11-02T08:39:56.455 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'90 2023-11-02T08:39:56.455 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'105 (29'1) modify 1:d858c035:::existing_1:head by client.4632.0:1 2023-11-02T08:37:39.710906+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:39:56.456 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 20 _merge_object_divergent_entries: keeping 29'105 (29'1) modify 1:d858c035:::existing_1:head by client.4632.0:1 2023-11-02T08:37:39.710906+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:39:56.456 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2023-11-02T08:39:56.456 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'105 last_divergent_update: 29'105 2023-11-02T08:39:56.456 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2023-11-02T08:39:56.456 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2023-11-02T08:39:56.457 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2023-11-02T08:39:56.457 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'105 (29'1) modify 1:d858c035:::existing_1:head by client.4632.0:1 2023-11-02T08:37:39.710906+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:39:56.457 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2023-11-02T08:39:56.457 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2023-11-02T08:39:56.457 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4626.0:1 2023-11-02T08:37:39.701797+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2023-11-02T08:39:56.457 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 20 _merge_object_divergent_entries: keeping 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4626.0:1 2023-11-02T08:37:39.701797+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2023-11-02T08:39:56.458 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2023-11-02T08:39:56.458 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'103 last_divergent_update: 29'103 2023-11-02T08:39:56.458 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2023-11-02T08:39:56.458 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:39:25.401+0000 7f0921cc2640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2023-11-02T08:39:56.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2023-11-02T08:39:56.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log 2023-11-02T08:39:56.460 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2023-11-02T08:39:56.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log 2023-11-02T08:39:56.461 INFO:tasks.workunit.client.0.smithi116.stdout:success 2023-11-02T08:39:56.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2023-11-02T08:39:56.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.85082.out 2023-11-02T08:39:56.463 INFO:tasks.workunit.client.0.smithi116.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.85082.out': No such file or directory 2023-11-02T08:39:56.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2023-11-02T08:39:56.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2023-11-02T08:39:56.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2023-11-02T08:39:57.031 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test' does not exist 2023-11-02T08:39:57.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2023-11-02T08:39:57.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:39:57.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:39:57.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:39:57.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:39:57.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:39:57.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:39:57.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2023-11-02T08:39:57.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2023-11-02T08:39:57.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T08:39:57.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2023-11-02T08:39:57.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:39:57.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:39:57.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:39:57.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:39:57.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:39:57.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:39:57.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T08:39:57.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T08:39:57.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T08:39:57.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T08:39:57.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T08:39:57.358 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T08:39:57.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:39:57.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T08:39:57.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T08:39:57.360 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:39:57.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T08:39:57.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T08:39:57.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2023-11-02T08:39:57.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T08:39:57.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:39:57.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:39:57.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.85082 2023-11-02T08:39:57.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T08:39:57.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T08:39:57.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2023-11-02T08:39:57.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2023-11-02T08:39:57.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2023-11-02T08:39:57.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2023-11-02T08:39:57.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2023-11-02T08:39:57.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T08:39:57.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2023-11-02T08:39:57.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:39:57.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:39:57.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:39:57.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:39:57.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:39:57.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:39:57.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T08:39:57.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T08:39:57.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T08:39:57.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T08:39:57.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T08:39:57.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T08:39:57.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:39:57.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T08:39:57.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T08:39:57.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:39:57.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T08:39:57.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T08:39:57.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2023-11-02T08:39:57.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T08:39:57.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:39:57.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:39:57.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.85082 2023-11-02T08:39:57.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T08:39:57.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T08:39:57.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2023-11-02T08:39:57.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T08:39:57.393 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:39:57.393 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:39:57.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.85082 2023-11-02T08:39:57.394 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T08:39:57.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-11-02T08:39:57.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T08:39:57.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2023-11-02T08:39:57.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2023-11-02T08:39:57.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2023-11-02T08:39:57.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2023-11-02T08:39:57.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2023-11-02T08:39:57.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2023-11-02T08:39:57.396 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2023-11-02T08:39:57.397 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2023-11-02T08:39:57.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2023-11-02T08:39:57.398 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:39:57.398 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:39:57.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2023-11-02T08:39:57.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2023-11-02T08:39:57.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T08:39:57.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T08:39:57.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T08:39:57.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2023-11-02T08:39:57.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2023-11-02T08:39:57.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T08:39:57.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:39:57.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:39:57.756 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:39:57.756 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:39:57.756 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:39:57.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:39:57.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T08:39:57.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T08:39:57.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T08:39:57.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T08:39:57.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T08:39:57.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T08:39:57.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T08:39:57.941 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T08:39:57.941 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T08:39:57.941 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T08:39:57.941 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:39:57.941 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:39:57.942 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:39:57.942 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-mon.a.asok 2023-11-02T08:39:57.942 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T08:39:57.942 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.85082/ceph-mon.a.asok config get fsid 2023-11-02T08:39:58.019 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T08:39:58.019 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T08:39:58.019 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T08:39:58.019 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T08:39:58.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T08:39:58.020 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T08:39:58.020 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T08:39:58.020 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T08:39:58.020 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:39:58.021 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:39:58.021 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:39:58.021 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-mon.a.asok 2023-11-02T08:39:58.021 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T08:39:58.021 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.85082/ceph-mon.a.asok config get mon_host 2023-11-02T08:39:58.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2023-11-02T08:39:58.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2023-11-02T08:39:58.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T08:39:58.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T08:39:58.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T08:39:58.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2023-11-02T08:39:58.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T08:39:58.944 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T08:39:58.944 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:39:58.944 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:39:58.944 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:39:58.945 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:39:58.945 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:39:58.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:39:58.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T08:39:58.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T08:39:58.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2023-11-02T08:39:58.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2023-11-02T08:39:58.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2023-11-02T08:39:58.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:39:58.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T08:39:58.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:39:58.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2023-11-02T08:39:58.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:39:58.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:39:58.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:39:58.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:39:58.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2023-11-02T08:39:58.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2023-11-02T08:39:58.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:39:58.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:39:58.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:39:58.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:39:58.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:39:58.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:39:58.982 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:39:58.982 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:39:58.983 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:39:58.983 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:39:58.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:39:58.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:39:58.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:39:58.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:39:58.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:39:58.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:39:58.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:39:58.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:39:58.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:39:58.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:39:58.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:39:58.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:39:58.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2023-11-02T08:39:58.986 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:39:58.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=584ee92b-5cfd-43f0-8535-9a70b0612348 2023-11-02T08:39:58.988 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 584ee92b-5cfd-43f0-8535-9a70b0612348 2023-11-02T08:39:58.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 584ee92b-5cfd-43f0-8535-9a70b0612348' 2023-11-02T08:39:58.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:39:59.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBeYENlpiaOOxAAa4jhr36cChmvYsm5g6GouA== 2023-11-02T08:39:59.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBeYENlpiaOOxAAa4jhr36cChmvYsm5g6GouA=="}' 2023-11-02T08:39:59.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 584ee92b-5cfd-43f0-8535-9a70b0612348 -i td/divergent-priors/0/new.json 2023-11-02T08:39:59.373 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:39:59.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2023-11-02T08:39:59.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBeYENlpiaOOxAAa4jhr36cChmvYsm5g6GouA== --osd-uuid 584ee92b-5cfd-43f0-8535-9a70b0612348 2023-11-02T08:39:59.418 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:39:59.415+0000 7f4816cfa500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:39:59.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:39:59.417+0000 7f4816cfa500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:39:59.421 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:39:59.418+0000 7f4816cfa500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:39:59.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:39:59.419+0000 7f4816cfa500 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2023-11-02T08:39:59.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:39:59.419+0000 7f4816cfa500 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2023-11-02T08:39:59.422 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:39:59.419+0000 7f4816cfa500 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2023-11-02T08:39:59.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:39:59.493+0000 7f4816cfa500 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2023-11-02T08:40:04.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2023-11-02T08:40:04.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:40:04.740 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T08:40:04.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T08:40:04.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:40:05.270 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T08:40:05.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T08:40:05.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:40:05.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:40:05.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:40:05.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:40:05.304 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:40:05.301+0000 7fcdb602a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:40:05.338 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:40:05.335+0000 7fcdb602a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:40:05.339 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:40:05.336+0000 7fcdb602a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:40:05.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T08:40:05.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:40:05.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:40:05.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:40:05.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:40:05.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:05.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:40:05.693 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:40:05.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:05.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:40:05.856 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:40:05.853+0000 7fcdb602a500 -1 Falling back to public interface 2023-11-02T08:40:06.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:07.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:07.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:07.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:40:07.070 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:40:07.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:07.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:40:07.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:08.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:40:08.143+0000 7fcdb602a500 -1 osd.0 0 log_to_monitors true 2023-11-02T08:40:08.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:08.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:08.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:40:08.451 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:40:08.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:08.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:40:08.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:09.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:09.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:09.849 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:40:09.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:40:09.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:09.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:40:10.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:11.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:11.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:11.246 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:40:11.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:40:11.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:11.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:40:11.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:12.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:12.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:12.738 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:40:12.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:40:12.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:12.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:40:13.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:14.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:14.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:14.250 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:40:14.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:40:14.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:14.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:40:14.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:15.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:15.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:15.758 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:40:15.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:40:15.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:15.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:40:16.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:17.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:17.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:17.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:40:17.179 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:40:17.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:17.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:40:17.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:18.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:18.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:18.575 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:40:18.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:40:18.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:18.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:40:19.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:20.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:20.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:20.023 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:40:20.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:40:20.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:20.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:40:20.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:21.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:21.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:21.725 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:40:21.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:40:21.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:21.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:40:22.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:23.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:23.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:23.247 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:40:23.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:40:23.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:23.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:40:23.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:24.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:24.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:24.876 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:40:24.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:40:24.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:24.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:40:25.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:26.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:26.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:26.279 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:40:26.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:40:26.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:26.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:40:27.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:28.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:28.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:28.386 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:40:28.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:40:28.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:28.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:40:29.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:30.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:30.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:30.004 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:40:30.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:40:30.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:30.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:40:30.431 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/112824,v1:127.0.0.1:6803/112824] [v2:127.0.0.1:6804/112824,v1:127.0.0.1:6805/112824] exists,up 584ee92b-5cfd-43f0-8535-9a70b0612348 2023-11-02T08:40:30.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:40:30.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:40:30.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:40:30.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2023-11-02T08:40:30.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2023-11-02T08:40:30.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2023-11-02T08:40:30.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:40:30.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-11-02T08:40:30.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:40:30.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2023-11-02T08:40:30.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:40:30.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:40:30.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:40:30.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:40:30.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2023-11-02T08:40:30.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2023-11-02T08:40:30.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:40:30.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:40:30.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:40:30.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:40:30.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:40:30.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:40:30.440 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:40:30.440 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:40:30.441 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:40:30.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:40:30.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:40:30.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:40:30.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:40:30.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:40:30.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:40:30.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:40:30.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:40:30.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:40:30.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:40:30.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:40:30.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:40:30.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:40:30.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2023-11-02T08:40:30.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:40:30.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4d681ad5-42d6-443f-ac67-86739e3ca916 2023-11-02T08:40:30.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4d681ad5-42d6-443f-ac67-86739e3ca916' 2023-11-02T08:40:30.445 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 4d681ad5-42d6-443f-ac67-86739e3ca916 2023-11-02T08:40:30.445 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:40:30.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB+YENl4DYeGxAAtxKL54e87+Yy7rZmK20TjQ== 2023-11-02T08:40:30.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB+YENl4DYeGxAAtxKL54e87+Yy7rZmK20TjQ=="}' 2023-11-02T08:40:30.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4d681ad5-42d6-443f-ac67-86739e3ca916 -i td/divergent-priors/1/new.json 2023-11-02T08:40:31.083 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:40:31.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2023-11-02T08:40:31.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB+YENl4DYeGxAAtxKL54e87+Yy7rZmK20TjQ== --osd-uuid 4d681ad5-42d6-443f-ac67-86739e3ca916 2023-11-02T08:40:31.129 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:40:31.126+0000 7f07d6214500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:40:31.131 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:40:31.128+0000 7f07d6214500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:40:31.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:40:31.129+0000 7f07d6214500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:40:31.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:40:31.129+0000 7f07d6214500 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2023-11-02T08:40:31.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:40:31.129+0000 7f07d6214500 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2023-11-02T08:40:31.133 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:40:31.129+0000 7f07d6214500 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2023-11-02T08:40:31.224 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:40:31.221+0000 7f07d6214500 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2023-11-02T08:40:36.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2023-11-02T08:40:36.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:40:36.706 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2023-11-02T08:40:36.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-11-02T08:40:36.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:40:37.259 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T08:40:37.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-11-02T08:40:37.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:40:37.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:40:37.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:40:37.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:40:37.293 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:40:37.290+0000 7f837001a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:40:37.295 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:40:37.292+0000 7f837001a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:40:37.296 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:40:37.293+0000 7f837001a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:40:37.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-11-02T08:40:37.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:40:37.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:40:37.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:40:37.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:40:37.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:37.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:40:37.662 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:40:37.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:37.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:40:37.825 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:40:37.822+0000 7f837001a500 -1 Falling back to public interface 2023-11-02T08:40:38.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:39.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:39.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:39.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:40:39.122 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:40:39.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:39.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:40:39.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:40.082 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:40:40.079+0000 7f837001a500 -1 osd.1 0 log_to_monitors true 2023-11-02T08:40:40.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:40.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:40.548 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:40:40.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:40:40.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:40.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:40:41.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:42.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:42.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:42.212 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:40:42.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:40:42.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:42.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:40:42.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:43.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:43.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:43.602 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:40:43.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:40:43.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:43.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:40:44.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:45.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:45.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:45.060 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:40:45.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:40:45.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:45.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:40:45.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:46.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:46.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:46.539 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:40:46.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:40:46.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:46.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:40:46.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:47.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:47.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:47.978 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:40:47.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:40:47.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:47.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:40:48.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:49.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:49.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:49.373 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:40:49.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:40:49.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:49.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:40:49.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:50.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:50.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:50.850 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:40:50.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:40:50.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:50.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:40:51.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:52.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:52.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:52.240 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:40:52.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:40:52.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:52.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:40:52.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:53.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:53.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:53.924 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:40:53.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:40:53.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:53.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:40:54.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:55.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:55.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:55.401 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:40:55.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:40:55.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:55.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:40:56.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:57.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:57.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:57.192 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:40:57.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:40:57.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:57.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:40:57.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:40:58.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:40:58.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:40:58.934 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:40:58.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:40:58.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:40:58.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:40:59.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:41:00.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:41:00.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:41:00.943 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:41:00.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:41:00.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:41:00.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:41:01.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:41:02.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:41:02.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:41:02.832 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:41:02.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:41:02.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:41:02.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:41:03.216 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/114539,v1:127.0.0.1:6811/114539] [v2:127.0.0.1:6812/114539,v1:127.0.0.1:6813/114539] exists,up 4d681ad5-42d6-443f-ac67-86739e3ca916 2023-11-02T08:41:03.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:41:03.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:41:03.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:41:03.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2023-11-02T08:41:03.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2023-11-02T08:41:03.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2023-11-02T08:41:03.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:41:03.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-11-02T08:41:03.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:41:03.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2023-11-02T08:41:03.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:41:03.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:41:03.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:41:03.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:41:03.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2023-11-02T08:41:03.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2023-11-02T08:41:03.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:41:03.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:41:03.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:41:03.224 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:41:03.225 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:41:03.225 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:41:03.225 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:41:03.225 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:41:03.225 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:41:03.226 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:41:03.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:41:03.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:41:03.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:41:03.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:41:03.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:41:03.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:41:03.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:41:03.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:41:03.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:41:03.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:41:03.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:41:03.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:41:03.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2023-11-02T08:41:03.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:41:03.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8b4576fe-bdf3-4241-80eb-7d5e8439887c 2023-11-02T08:41:03.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 8b4576fe-bdf3-4241-80eb-7d5e8439887c' 2023-11-02T08:41:03.229 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 8b4576fe-bdf3-4241-80eb-7d5e8439887c 2023-11-02T08:41:03.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:41:03.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCfYENlQY9KDhAAHbOJoFPX91jvZlzWHg3S+A== 2023-11-02T08:41:03.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCfYENlQY9KDhAAHbOJoFPX91jvZlzWHg3S+A=="}' 2023-11-02T08:41:03.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8b4576fe-bdf3-4241-80eb-7d5e8439887c -i td/divergent-priors/2/new.json 2023-11-02T08:41:03.808 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:41:03.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2023-11-02T08:41:03.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCfYENlQY9KDhAAHbOJoFPX91jvZlzWHg3S+A== --osd-uuid 8b4576fe-bdf3-4241-80eb-7d5e8439887c 2023-11-02T08:41:03.855 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:41:03.851+0000 7f5fb3612500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:41:03.877 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:41:03.874+0000 7f5fb3612500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:41:03.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:41:03.875+0000 7f5fb3612500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:41:03.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:41:03.875+0000 7f5fb3612500 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2023-11-02T08:41:03.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:41:03.875+0000 7f5fb3612500 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2023-11-02T08:41:03.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:41:03.875+0000 7f5fb3612500 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2023-11-02T08:41:03.917 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:41:03.914+0000 7f5fb3612500 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2023-11-02T08:41:09.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2023-11-02T08:41:09.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:41:09.463 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2023-11-02T08:41:09.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-11-02T08:41:09.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:41:09.990 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T08:41:09.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-11-02T08:41:09.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:41:09.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:41:09.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:41:09.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:41:10.025 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:41:10.022+0000 7f749a41f500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:41:10.030 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:41:10.027+0000 7f749a41f500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:41:10.031 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:41:10.028+0000 7f749a41f500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:41:10.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-11-02T08:41:10.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:41:10.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T08:41:10.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:41:10.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:41:10.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:41:10.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:41:10.429 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:41:10.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:41:10.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:41:10.560 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:41:10.557+0000 7f749a41f500 -1 Falling back to public interface 2023-11-02T08:41:10.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:41:11.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:41:11.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:41:11.846 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:41:11.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:41:11.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:41:11.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:41:12.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:41:12.841 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:41:12.838+0000 7f749a41f500 -1 osd.2 0 log_to_monitors true 2023-11-02T08:41:13.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:41:13.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:41:13.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:41:13.237 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:41:13.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:41:13.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:41:13.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:41:14.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:41:14.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:41:14.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:41:14.654 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:41:14.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:41:14.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:41:15.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:41:16.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:41:16.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:41:16.083 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:41:16.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:41:16.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:41:16.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:41:16.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:41:17.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:41:17.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:41:17.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:41:17.474 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:41:17.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:41:17.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:41:17.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:41:18.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:41:18.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:41:18.962 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:41:18.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:41:18.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:41:18.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:41:19.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:41:20.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:41:20.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:41:20.427 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:41:20.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:41:20.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:41:20.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:41:20.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:41:21.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:41:21.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:41:21.937 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:41:21.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:41:21.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:41:21.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:41:22.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:41:23.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:41:23.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:41:23.347 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:41:23.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:41:23.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:41:23.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:41:23.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:41:24.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:41:24.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:41:24.778 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:41:24.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:41:24.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:41:24.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:41:25.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:41:26.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:41:26.883 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:41:26.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:41:26.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:41:26.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:41:26.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:41:27.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:41:28.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:41:28.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:41:28.682 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:41:28.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:41:28.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:41:28.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:41:29.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:41:30.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:41:30.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:41:30.991 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:41:30.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:41:30.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:41:30.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:41:32.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:41:33.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:41:33.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:41:33.186 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:41:33.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:41:33.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:41:33.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:41:34.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:41:35.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:41:35.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:41:35.340 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:41:35.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:41:35.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:41:35.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:41:35.737 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/116252,v1:127.0.0.1:6819/116252] [v2:127.0.0.1:6820/116252,v1:127.0.0.1:6821/116252] exists,up 8b4576fe-bdf3-4241-80eb-7d5e8439887c 2023-11-02T08:41:35.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:41:35.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:41:35.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:41:35.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2023-11-02T08:41:36.509 INFO:tasks.workunit.client.0.smithi116.stderr:noout is set 2023-11-02T08:41:36.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2023-11-02T08:41:37.229 INFO:tasks.workunit.client.0.smithi116.stderr:noin is set 2023-11-02T08:41:37.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2023-11-02T08:41:38.132 INFO:tasks.workunit.client.0.smithi116.stderr:nodown is set 2023-11-02T08:41:38.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2023-11-02T08:41:38.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2023-11-02T08:41:38.978 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test' already exists 2023-11-02T08:41:38.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T08:41:39.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2023-11-02T08:41:40.626 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 size to 3 2023-11-02T08:41:40.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2023-11-02T08:41:41.380 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 min_size to 2 2023-11-02T08:41:41.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2023-11-02T08:41:42.090 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 pg_autoscale_mode to on 2023-11-02T08:41:42.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: divergent=-1 2023-11-02T08:41:42.102 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: date +%s 2023-11-02T08:41:42.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: start_time=1698914502 2023-11-02T08:41:42.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:655: TEST_divergent_3: max_duration=300 2023-11-02T08:41:42.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' -1 -le -1 ']' 2023-11-02T08:41:42.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: flush_pg_stats 2023-11-02T08:41:42.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:41:42.105 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:41:42.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:41:42.531 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:41:42.531 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:41:42.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:41:42.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:41:42.532 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:41:42.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836502 2023-11-02T08:41:42.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836502 2023-11-02T08:41:42.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836502' 2023-11-02T08:41:42.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:41:42.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:41:42.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672976 2023-11-02T08:41:42.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672976 2023-11-02T08:41:42.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836502 1-42949672976' 2023-11-02T08:41:42.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:41:42.766 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:41:42.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509450 2023-11-02T08:41:42.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509450 2023-11-02T08:41:42.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836502 1-42949672976 2-64424509450' 2023-11-02T08:41:42.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:41:42.886 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836502 2023-11-02T08:41:42.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:41:42.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:41:42.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836502 2023-11-02T08:41:42.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:41:42.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836502 2023-11-02T08:41:42.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2023-11-02T08:41:42.889 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836502 2023-11-02T08:41:42.889 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:41:43.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836502 -lt 21474836502 2023-11-02T08:41:43.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:41:43.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672976 2023-11-02T08:41:43.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:41:43.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T08:41:43.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672976 2023-11-02T08:41:43.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:41:43.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672976 2023-11-02T08:41:43.301 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672976 2023-11-02T08:41:43.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2023-11-02T08:41:43.301 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T08:41:43.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672976 -lt 42949672976 2023-11-02T08:41:43.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:41:43.676 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509450 2023-11-02T08:41:43.676 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:41:43.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:41:43.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509450 2023-11-02T08:41:43.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:41:43.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509450 2023-11-02T08:41:43.679 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509450 2023-11-02T08:41:43.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2023-11-02T08:41:43.679 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:41:44.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509448 -lt 64424509450 2023-11-02T08:41:44.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T08:41:45.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T08:41:45.101 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:41:45.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509450 -lt 64424509450 2023-11-02T08:41:45.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: wait_for_clean 2023-11-02T08:41:45.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T08:41:45.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T08:41:45.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T08:41:45.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-11-02T08:41:45.716 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T08:41:45.716 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T08:41:45.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T08:41:45.717 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T08:41:45.717 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T08:41:45.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-11-02T08:41:45.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T08:41:45.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T08:41:45.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T08:41:45.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:41:45.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:41:46.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:41:46.021 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:41:46.022 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:41:46.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:41:46.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:41:46.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:41:46.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836505 2023-11-02T08:41:46.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836505 2023-11-02T08:41:46.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836505' 2023-11-02T08:41:46.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:41:46.139 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:41:46.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672979 2023-11-02T08:41:46.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672979 2023-11-02T08:41:46.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836505 1-42949672979' 2023-11-02T08:41:46.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:41:46.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:41:46.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509453 2023-11-02T08:41:46.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509453 2023-11-02T08:41:46.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836505 1-42949672979 2-64424509453' 2023-11-02T08:41:46.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:41:46.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836505 2023-11-02T08:41:46.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:41:46.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:41:46.379 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836505 2023-11-02T08:41:46.379 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:41:46.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836505 2023-11-02T08:41:46.380 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836505 2023-11-02T08:41:46.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2023-11-02T08:41:46.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:41:46.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836502 -lt 21474836505 2023-11-02T08:41:46.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T08:41:47.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T08:41:47.753 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:41:48.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836505 -lt 21474836505 2023-11-02T08:41:48.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:41:48.185 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672979 2023-11-02T08:41:48.185 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:41:48.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T08:41:48.186 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672979 2023-11-02T08:41:48.186 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:41:48.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672979 2023-11-02T08:41:48.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2023-11-02T08:41:48.187 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672979 2023-11-02T08:41:48.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T08:41:48.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672979 -lt 42949672979 2023-11-02T08:41:48.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:41:48.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509453 2023-11-02T08:41:48.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:41:48.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:41:48.564 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509453 2023-11-02T08:41:48.565 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:41:48.565 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509453 2023-11-02T08:41:48.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509453 2023-11-02T08:41:48.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2023-11-02T08:41:48.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:41:48.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509453 -lt 64424509453 2023-11-02T08:41:48.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T08:41:48.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:41:48.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:41:49.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2023-11-02T08:41:49.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:41:49.452 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:41:49.452 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:41:49.452 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:41:49.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:41:49.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:41:49.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:41:49.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T08:41:49.841 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:41:49.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:41:49.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:41:50.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2023-11-02T08:41:50.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T08:41:50.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T08:41:50.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: ceph pg dump pgs --format=json 2023-11-02T08:41:50.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2023-11-02T08:41:50.746 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:41:50.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: divergent=1 2023-11-02T08:41:50.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2023-11-02T08:41:50.758 INFO:tasks.workunit.client.0.smithi116.stdout:primary and soon to be divergent is 1 2023-11-02T08:41:50.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:665: TEST_divergent_3: ceph pg dump pgs 2023-11-02T08:41:51.115 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:41:51.117 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-02T08:41:39.553340+0000 0'0 29:14 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 periodic scrub scheduled @ 2023-11-03T15:54:05.988549+0000 0 0 2023-11-02T08:41:51.118 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:41:51.118 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:41:51.118 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:41:51.126 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: date +%s 2023-11-02T08:41:51.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: current_time=1698914511 2023-11-02T08:41:51.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: expr 1698914511 - 1698914502 2023-11-02T08:41:51.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: elapsed_time=9 2023-11-02T08:41:51.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: '[' 9 -gt 300 ']' 2023-11-02T08:41:51.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' 1 -le -1 ']' 2023-11-02T08:41:51.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:675: TEST_divergent_3: local non_divergent= 2023-11-02T08:41:51.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2023-11-02T08:41:51.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 0 = 1 ']' 2023-11-02T08:41:51.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0' 2023-11-02T08:41:51.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2023-11-02T08:41:51.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 1 = 1 ']' 2023-11-02T08:41:51.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:679: TEST_divergent_3: continue 2023-11-02T08:41:51.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2023-11-02T08:41:51.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 2 = 1 ']' 2023-11-02T08:41:51.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0 2' 2023-11-02T08:41:51.131 INFO:tasks.workunit.client.0.smithi116.stdout:writing initial objects 2023-11-02T08:41:51.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:684: TEST_divergent_3: echo 'writing initial objects' 2023-11-02T08:41:51.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: seq 1 100 2023-11-02T08:41:51.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:51.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2023-11-02T08:41:51.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:51.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2023-11-02T08:41:51.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:51.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2023-11-02T08:41:51.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:51.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2023-11-02T08:41:51.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:51.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2023-11-02T08:41:51.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:51.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2023-11-02T08:41:51.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:51.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2023-11-02T08:41:52.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:52.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2023-11-02T08:41:52.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:52.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2023-11-02T08:41:52.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:52.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2023-11-02T08:41:52.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:52.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2023-11-02T08:41:52.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:52.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2023-11-02T08:41:52.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:52.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2023-11-02T08:41:53.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:53.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2023-11-02T08:41:53.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:53.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2023-11-02T08:41:53.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:53.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2023-11-02T08:41:53.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:53.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2023-11-02T08:41:53.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:53.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2023-11-02T08:41:53.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:53.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2023-11-02T08:41:53.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:53.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2023-11-02T08:41:54.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:54.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2023-11-02T08:41:54.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:54.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2023-11-02T08:41:54.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:54.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2023-11-02T08:41:54.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:54.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2023-11-02T08:41:54.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:54.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2023-11-02T08:41:54.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:54.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2023-11-02T08:41:54.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:54.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2023-11-02T08:41:54.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:54.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2023-11-02T08:41:55.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:55.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2023-11-02T08:41:55.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:55.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2023-11-02T08:41:55.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:55.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2023-11-02T08:41:55.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:55.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2023-11-02T08:41:55.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:55.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2023-11-02T08:41:55.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:55.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2023-11-02T08:41:56.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:56.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2023-11-02T08:41:56.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:56.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2023-11-02T08:41:56.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:56.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2023-11-02T08:41:56.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:56.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2023-11-02T08:41:56.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:56.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2023-11-02T08:41:56.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:56.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2023-11-02T08:41:56.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:56.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2023-11-02T08:41:56.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:56.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2023-11-02T08:41:57.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:57.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2023-11-02T08:41:57.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:57.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2023-11-02T08:41:57.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:57.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2023-11-02T08:41:57.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:57.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2023-11-02T08:41:57.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:57.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2023-11-02T08:41:57.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:57.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2023-11-02T08:41:57.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:57.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2023-11-02T08:41:58.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:58.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2023-11-02T08:41:58.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:58.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2023-11-02T08:41:58.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:58.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2023-11-02T08:41:58.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:58.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2023-11-02T08:41:58.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:58.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2023-11-02T08:41:58.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:58.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2023-11-02T08:41:58.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:58.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2023-11-02T08:41:59.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:59.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2023-11-02T08:41:59.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:59.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2023-11-02T08:41:59.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:59.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2023-11-02T08:41:59.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:59.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2023-11-02T08:41:59.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:59.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2023-11-02T08:41:59.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:59.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2023-11-02T08:41:59.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:41:59.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2023-11-02T08:42:00.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:00.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2023-11-02T08:42:00.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:00.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2023-11-02T08:42:00.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:00.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2023-11-02T08:42:00.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:00.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2023-11-02T08:42:00.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:00.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2023-11-02T08:42:00.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:00.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2023-11-02T08:42:00.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:00.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2023-11-02T08:42:00.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:00.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2023-11-02T08:42:01.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:01.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2023-11-02T08:42:01.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:01.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2023-11-02T08:42:01.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:01.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2023-11-02T08:42:01.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:01.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2023-11-02T08:42:01.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:01.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2023-11-02T08:42:01.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:01.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2023-11-02T08:42:01.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:01.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2023-11-02T08:42:02.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:02.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2023-11-02T08:42:02.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:02.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2023-11-02T08:42:02.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:02.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2023-11-02T08:42:02.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:02.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2023-11-02T08:42:02.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:02.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2023-11-02T08:42:03.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:03.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2023-11-02T08:42:03.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:03.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2023-11-02T08:42:03.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:03.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2023-11-02T08:42:03.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:03.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2023-11-02T08:42:03.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:03.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2023-11-02T08:42:03.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:03.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2023-11-02T08:42:04.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:04.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2023-11-02T08:42:04.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:04.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2023-11-02T08:42:04.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:04.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2023-11-02T08:42:04.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:04.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2023-11-02T08:42:04.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:04.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2023-11-02T08:42:04.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:04.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2023-11-02T08:42:04.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:04.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2023-11-02T08:42:05.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:05.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2023-11-02T08:42:05.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:05.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2023-11-02T08:42:06.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:06.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2023-11-02T08:42:06.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2023-11-02T08:42:06.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2023-11-02T08:42:06.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2023-11-02T08:42:06.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: wait_for_clean 2023-11-02T08:42:06.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T08:42:06.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T08:42:06.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T08:42:06.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2023-11-02T08:42:06.708 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T08:42:06.708 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T08:42:06.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T08:42:06.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T08:42:06.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T08:42:06.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2023-11-02T08:42:06.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T08:42:06.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T08:42:06.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T08:42:06.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:42:06.762 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:42:07.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:42:07.163 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:42:07.163 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:42:07.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:42:07.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:42:07.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:42:07.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836511 2023-11-02T08:42:07.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836511 2023-11-02T08:42:07.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836511' 2023-11-02T08:42:07.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:42:07.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:42:07.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672985 2023-11-02T08:42:07.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672985 2023-11-02T08:42:07.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836511 1-42949672985' 2023-11-02T08:42:07.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:42:07.403 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:42:07.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509459 2023-11-02T08:42:07.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509459 2023-11-02T08:42:07.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836511 1-42949672985 2-64424509459' 2023-11-02T08:42:07.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:42:07.522 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836511 2023-11-02T08:42:07.522 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:42:07.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:42:07.523 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836511 2023-11-02T08:42:07.524 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:42:07.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836511 2023-11-02T08:42:07.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2023-11-02T08:42:07.525 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836511 2023-11-02T08:42:07.525 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:42:07.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836509 -lt 21474836511 2023-11-02T08:42:07.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T08:42:08.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T08:42:08.918 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:42:09.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836511 -lt 21474836511 2023-11-02T08:42:09.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:42:09.558 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672985 2023-11-02T08:42:09.558 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:42:09.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T08:42:09.559 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672985 2023-11-02T08:42:09.559 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:42:09.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672985 2023-11-02T08:42:09.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2023-11-02T08:42:09.560 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672985 2023-11-02T08:42:09.561 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T08:42:10.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672986 -lt 42949672985 2023-11-02T08:42:10.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:42:10.144 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509459 2023-11-02T08:42:10.145 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:42:10.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:42:10.146 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509459 2023-11-02T08:42:10.146 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:42:10.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509459 2023-11-02T08:42:10.147 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509459 2023-11-02T08:42:10.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2023-11-02T08:42:10.148 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:42:10.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509459 -lt 64424509459 2023-11-02T08:42:10.537 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T08:42:10.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:42:10.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:42:11.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2023-11-02T08:42:11.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:42:11.042 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:42:11.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:42:11.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:42:11.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:42:11.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:42:11.044 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:42:11.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2023-11-02T08:42:11.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:42:11.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:42:11.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:42:11.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2023-11-02T08:42:11.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T08:42:11.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T08:42:11.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: get_pg test existing_1 2023-11-02T08:42:11.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2023-11-02T08:42:11.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2023-11-02T08:42:11.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2023-11-02T08:42:11.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2023-11-02T08:42:12.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: local pgid=1.1b 2023-11-02T08:42:12.511 INFO:tasks.workunit.client.0.smithi116.stdout:blackholing osds 0 2 2023-11-02T08:42:12.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: echo 'blackholing osds 0 2' 2023-11-02T08:42:12.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:697: TEST_divergent_3: ceph pg dump pgs 2023-11-02T08:42:12.924 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:42:12.924 INFO:tasks.workunit.client.0.smithi116.stdout:1.1f 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:09.854502+0000 32'97 36:122 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.924 INFO:tasks.workunit.client.0.smithi116.stdout:1.1e 4 0 0 0 0 1947 0 0 14 84 14 active+clean 2023-11-02T08:42:06.228694+0000 34'98 34:125 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.925 INFO:tasks.workunit.client.0.smithi116.stdout:1.1d 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:09.853785+0000 32'97 36:122 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.925 INFO:tasks.workunit.client.0.smithi116.stdout:1.b 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:09.854316+0000 32'97 36:122 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.925 INFO:tasks.workunit.client.0.smithi116.stdout:1.a 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:06.227971+0000 32'97 34:124 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.925 INFO:tasks.workunit.client.0.smithi116.stdout:1.9 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:09.853448+0000 32'97 36:122 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.925 INFO:tasks.workunit.client.0.smithi116.stdout:1.8 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:06.228578+0000 32'97 34:124 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.926 INFO:tasks.workunit.client.0.smithi116.stdout:1.7 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:09.854594+0000 32'97 36:122 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.926 INFO:tasks.workunit.client.0.smithi116.stdout:1.6 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:06.225077+0000 32'97 34:124 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.926 INFO:tasks.workunit.client.0.smithi116.stdout:1.5 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:09.853054+0000 32'97 36:122 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.926 INFO:tasks.workunit.client.0.smithi116.stdout:1.4 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:06.224887+0000 32'97 34:124 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.926 INFO:tasks.workunit.client.0.smithi116.stdout:1.3 4 0 0 0 0 1948 0 0 14 84 14 active+clean 2023-11-02T08:42:09.696398+0000 34'98 36:134 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.927 INFO:tasks.workunit.client.0.smithi116.stdout:1.2 3 0 0 0 0 1465 0 0 13 84 13 active+clean 2023-11-02T08:42:06.225105+0000 32'97 34:124 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.927 INFO:tasks.workunit.client.0.smithi116.stdout:1.c 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:06.229158+0000 32'97 34:124 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.927 INFO:tasks.workunit.client.0.smithi116.stdout:1.1 2 0 0 0 0 966 0 0 13 84 13 active+clean 2023-11-02T08:42:09.981554+0000 32'97 36:134 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:42:09.981464+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T13:24:58.648214+0000 2 0 2023-11-02T08:42:12.927 INFO:tasks.workunit.client.0.smithi116.stdout:1.1c 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:06.229416+0000 32'97 34:124 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.927 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 3 0 0 0 0 1449 0 0 13 84 13 active+clean 2023-11-02T08:42:07.198418+0000 32'97 34:136 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:07.198292+0000 32'97 2023-11-02T08:42:07.198292+0000 0 1 periodic scrub scheduled @ 2023-11-03T16:56:03.107625+0000 3 0 2023-11-02T08:42:12.928 INFO:tasks.workunit.client.0.smithi116.stdout:1.d 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:09.851671+0000 32'97 36:122 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.928 INFO:tasks.workunit.client.0.smithi116.stdout:1.e 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:06.227572+0000 32'97 34:124 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.928 INFO:tasks.workunit.client.0.smithi116.stdout:1.f 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:09.852444+0000 32'97 36:122 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.928 INFO:tasks.workunit.client.0.smithi116.stdout:1.10 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:06.230332+0000 32'97 34:124 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.928 INFO:tasks.workunit.client.0.smithi116.stdout:1.11 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:09.854410+0000 32'97 36:122 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.929 INFO:tasks.workunit.client.0.smithi116.stdout:1.12 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:06.229233+0000 32'97 34:124 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.929 INFO:tasks.workunit.client.0.smithi116.stdout:1.13 3 0 0 0 0 1449 0 0 13 84 13 active+clean 2023-11-02T08:42:10.923351+0000 32'97 36:134 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:42:10.923269+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T16:29:09.218443+0000 3 0 2023-11-02T08:42:12.929 INFO:tasks.workunit.client.0.smithi116.stdout:1.14 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:06.228338+0000 32'97 34:124 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.929 INFO:tasks.workunit.client.0.smithi116.stdout:1.15 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:09.854134+0000 32'97 36:122 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.929 INFO:tasks.workunit.client.0.smithi116.stdout:1.16 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:06.229569+0000 32'97 34:124 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.930 INFO:tasks.workunit.client.0.smithi116.stdout:1.17 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:09.696636+0000 32'97 36:122 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.930 INFO:tasks.workunit.client.0.smithi116.stdout:1.18 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:06.228494+0000 32'97 34:124 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.930 INFO:tasks.workunit.client.0.smithi116.stdout:1.19 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:09.853240+0000 32'97 36:122 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.930 INFO:tasks.workunit.client.0.smithi116.stdout:1.1a 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:06.230812+0000 32'97 34:124 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.930 INFO:tasks.workunit.client.0.smithi116.stdout:1.1b 4 0 0 0 0 1947 0 0 14 84 14 active+clean 2023-11-02T08:42:09.854882+0000 34'98 36:133 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:12.931 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:42:12.931 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:42:12.931 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:42:12.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2023-11-02T08:42:12.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.0 2023-11-02T08:42:12.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-11-02T08:42:12.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-11-02T08:42:12.937 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:42:12.937 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:42:12.937 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:42:12.937 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.0.asok 2023-11-02T08:42:12.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2023-11-02T08:42:12.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.0.asok config set objectstore_blackhole 1 2023-11-02T08:42:13.014 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T08:42:13.014 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2023-11-02T08:42:13.014 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T08:42:13.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2023-11-02T08:42:13.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.2 2023-11-02T08:42:13.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-11-02T08:42:13.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-11-02T08:42:13.023 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:42:13.023 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:42:13.023 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:42:13.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.2.asok 2023-11-02T08:42:13.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2023-11-02T08:42:13.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.2.asok config set objectstore_blackhole 1 2023-11-02T08:42:13.100 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T08:42:13.101 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2023-11-02T08:42:13.101 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T08:42:13.108 INFO:tasks.workunit.client.0.smithi116.stdout:create new divergent objects 2023-11-02T08:42:13.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: echo 'create new divergent objects' 2023-11-02T08:42:13.108 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: seq 1 2 2023-11-02T08:42:13.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2023-11-02T08:42:13.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2023-11-02T08:42:13.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_1 2023-11-02T08:42:13.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:710: TEST_divergent_3: echo 'writing divergent objects' 2023-11-02T08:42:13.110 INFO:tasks.workunit.client.0.smithi116.stdout:writing divergent objects 2023-11-02T08:42:13.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_2 2023-11-02T08:42:13.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: seq 1 2 2023-11-02T08:42:13.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2023-11-02T08:42:13.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2023-11-02T08:42:13.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2023-11-02T08:42:13.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: echo 'remove divergent objects' 2023-11-02T08:42:13.113 INFO:tasks.workunit.client.0.smithi116.stdout:remove divergent objects 2023-11-02T08:42:13.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2023-11-02T08:42:13.114 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: seq 1 2 2023-11-02T08:42:13.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2023-11-02T08:42:13.114 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 1 + 2 2023-11-02T08:42:13.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=3 2023-11-02T08:42:13.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2023-11-02T08:42:13.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_3 2023-11-02T08:42:13.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 2 + 2 2023-11-02T08:42:13.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=4 2023-11-02T08:42:13.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:722: TEST_divergent_3: sleep 10 2023-11-02T08:42:13.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_4 2023-11-02T08:42:23.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:723: TEST_divergent_3: killall -9 rados 2023-11-02T08:42:23.120 INFO:tasks.workunit.client.0.smithi116.stderr:rados: no process found 2023-11-02T08:42:23.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: echo 'killing all the osds' 2023-11-02T08:42:23.120 INFO:tasks.workunit.client.0.smithi116.stdout:killing all the osds 2023-11-02T08:42:23.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:727: TEST_divergent_3: ceph pg dump pgs 2023-11-02T08:42:23.491 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:42:23.492 INFO:tasks.workunit.client.0.smithi116.stdout:1.1f 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:13.658249+0000 32'97 40:134 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:23.492 INFO:tasks.workunit.client.0.smithi116.stdout:1.1e 4 0 0 0 0 1947 0 0 14 84 14 active+clean 2023-11-02T08:42:12.033287+0000 34'98 40:137 [0,1,2] 0 [0,1,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:23.492 INFO:tasks.workunit.client.0.smithi116.stdout:1.1d 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:09.853785+0000 32'97 41:128 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:23.492 INFO:tasks.workunit.client.0.smithi116.stdout:1.b 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:13.658019+0000 32'97 40:134 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:23.492 INFO:tasks.workunit.client.0.smithi116.stdout:1.a 3 0 0 0 0 1449 0 0 13 84 13 active+clean 2023-11-02T08:42:20.387043+0000 32'97 41:137 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:42:20.386938+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T20:11:23.199988+0000 3 0 2023-11-02T08:42:23.493 INFO:tasks.workunit.client.0.smithi116.stdout:1.9 3 0 0 0 0 1111 0 0 14 84 14 active+clean 2023-11-02T08:42:09.853448+0000 39'98 41:129 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:23.493 INFO:tasks.workunit.client.0.smithi116.stdout:1.8 3 0 0 0 0 1449 0 0 13 84 13 active+clean 2023-11-02T08:42:17.109406+0000 32'97 40:142 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:17.109289+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T08:57:51.194607+0000 3 0 2023-11-02T08:42:23.493 INFO:tasks.workunit.client.0.smithi116.stdout:1.7 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:13.657784+0000 32'97 40:134 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:23.493 INFO:tasks.workunit.client.0.smithi116.stdout:1.6 4 0 0 0 0 1932 0 0 13 84 13 active+clean 2023-11-02T08:42:19.366709+0000 32'97 41:137 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:42:19.366599+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T11:09:38.248352+0000 4 0 2023-11-02T08:42:23.493 INFO:tasks.workunit.client.0.smithi116.stdout:1.5 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:09.853054+0000 32'97 41:128 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:23.494 INFO:tasks.workunit.client.0.smithi116.stdout:1.4 2 0 0 0 0 966 0 0 13 84 13 active+clean 2023-11-02T08:42:18.171223+0000 32'97 40:142 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:18.171099+0000 32'97 2023-11-02T08:42:18.171099+0000 0 1 periodic scrub scheduled @ 2023-11-03T16:21:33.552952+0000 2 0 2023-11-02T08:42:23.494 INFO:tasks.workunit.client.0.smithi116.stdout:1.3 4 0 0 0 0 1948 0 0 14 84 14 active+clean 2023-11-02T08:42:13.658020+0000 34'98 40:135 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:23.494 INFO:tasks.workunit.client.0.smithi116.stdout:1.2 1 0 0 0 0 483 0 0 13 84 13 active+clean 2023-11-02T08:42:12.415782+0000 32'97 40:137 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:42:12.415700+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T10:44:28.963015+0000 1 0 2023-11-02T08:42:23.495 INFO:tasks.workunit.client.0.smithi116.stdout:1.c 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:06.229158+0000 32'97 39:130 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:23.495 INFO:tasks.workunit.client.0.smithi116.stdout:1.1 2 0 0 0 0 966 0 0 13 84 13 active+clean 2023-11-02T08:42:09.981554+0000 32'97 41:140 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:42:09.981464+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T13:24:58.648214+0000 2 0 2023-11-02T08:42:23.495 INFO:tasks.workunit.client.0.smithi116.stdout:1.1c 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:06.229416+0000 32'97 39:130 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:23.495 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 3 0 0 0 0 1449 0 0 13 84 13 active+clean 2023-11-02T08:42:07.198418+0000 32'97 39:142 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:07.198292+0000 32'97 2023-11-02T08:42:07.198292+0000 0 1 periodic scrub scheduled @ 2023-11-03T16:56:03.107625+0000 3 0 2023-11-02T08:42:23.495 INFO:tasks.workunit.client.0.smithi116.stdout:1.d 3 0 0 0 0 1449 0 0 13 84 13 active+clean 2023-11-02T08:42:19.844826+0000 32'97 41:140 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:42:19.844682+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T12:02:33.908652+0000 3 0 2023-11-02T08:42:23.496 INFO:tasks.workunit.client.0.smithi116.stdout:1.e 3 0 0 0 0 981 0 0 15 84 15 active+clean 2023-11-02T08:42:12.033008+0000 39'99 40:127 [0,1,2] 0 [0,1,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:23.496 INFO:tasks.workunit.client.0.smithi116.stdout:1.f 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:13.657690+0000 32'97 40:134 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:23.496 INFO:tasks.workunit.client.0.smithi116.stdout:1.10 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:06.230332+0000 32'97 39:130 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:23.497 INFO:tasks.workunit.client.0.smithi116.stdout:1.11 2 0 0 0 0 966 0 0 14 84 14 active+clean 2023-11-02T08:42:18.867262+0000 39'98 41:141 [2,0,1] 2 [2,0,1] 2 39'98 2023-11-02T08:42:18.867158+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T09:47:05.817612+0000 2 0 2023-11-02T08:42:23.497 INFO:tasks.workunit.client.0.smithi116.stdout:1.12 1 0 0 0 0 483 0 0 13 84 13 active+clean 2023-11-02T08:42:15.453233+0000 32'97 40:137 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:42:15.453129+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T20:04:33.681148+0000 1 0 2023-11-02T08:42:23.497 INFO:tasks.workunit.client.0.smithi116.stdout:1.13 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:13.582328+0000 32'97 40:134 [1,2,0] 1 [1,2,0] 1 32'97 2023-11-02T08:42:10.923269+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:23.497 INFO:tasks.workunit.client.0.smithi116.stdout:1.14 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:06.228338+0000 32'97 39:130 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:23.497 INFO:tasks.workunit.client.0.smithi116.stdout:1.15 3 0 0 0 0 1449 0 0 13 84 13 active+clean 2023-11-02T08:42:14.857520+0000 32'97 41:140 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:42:14.857410+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T13:02:43.711974+0000 3 0 2023-11-02T08:42:23.497 INFO:tasks.workunit.client.0.smithi116.stdout:1.16 4 0 0 0 0 1932 0 0 13 84 13 active+clean 2023-11-02T08:42:16.428638+0000 32'97 40:137 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:42:16.428517+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T15:31:05.948447+0000 4 0 2023-11-02T08:42:23.498 INFO:tasks.workunit.client.0.smithi116.stdout:1.17 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:13.658218+0000 32'97 40:134 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:23.498 INFO:tasks.workunit.client.0.smithi116.stdout:1.18 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:06.228494+0000 32'97 39:130 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:23.498 INFO:tasks.workunit.client.0.smithi116.stdout:1.19 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:09.853240+0000 32'97 41:128 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:23.498 INFO:tasks.workunit.client.0.smithi116.stdout:1.1a 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:42:12.033689+0000 32'97 40:125 [0,1,2] 0 [0,1,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:23.499 INFO:tasks.workunit.client.0.smithi116.stdout:1.1b 5 0 0 0 0 1594 0 0 16 84 16 active+clean 2023-11-02T08:42:13.582327+0000 40'100 40:137 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:23.499 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:42:23.499 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:42:23.499 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:42:23.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2023-11-02T08:42:23.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:42:23.503 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:42:23.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:42:23.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:42:23.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:42:23.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:42:23.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2023-11-02T08:42:23.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.0 2023-11-02T08:42:24.124 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 is already down. 2023-11-02T08:42:24.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2023-11-02T08:42:24.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.1 2023-11-02T08:42:24.664 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 is already down. 2023-11-02T08:42:24.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2023-11-02T08:42:24.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.2 2023-11-02T08:42:25.359 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 is already down. 2023-11-02T08:42:25.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2023-11-02T08:42:25.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.0 2023-11-02T08:42:25.895 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 is already out. 2023-11-02T08:42:25.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2023-11-02T08:42:25.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.2 2023-11-02T08:42:26.453 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 is already out. 2023-11-02T08:42:26.464 INFO:tasks.workunit.client.0.smithi116.stdout:bringing up non_divergent 0 2 2023-11-02T08:42:26.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2023-11-02T08:42:26.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2023-11-02T08:42:26.855 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:42:26.855 INFO:tasks.workunit.client.0.smithi116.stdout:1.1f 3 0 0 0 0 1464 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:13.658249+0000 32'97 40:134 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:26.856 INFO:tasks.workunit.client.0.smithi116.stdout:1.1e 4 0 0 0 0 1947 0 0 14 84 14 stale+active+clean 2023-11-02T08:42:12.033287+0000 34'98 40:137 [0,1,2] 0 [0,1,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:26.856 INFO:tasks.workunit.client.0.smithi116.stdout:1.1d 3 0 0 0 0 1464 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:09.853785+0000 32'97 41:128 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:26.856 INFO:tasks.workunit.client.0.smithi116.stdout:1.b 3 0 0 0 0 1464 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:13.658019+0000 32'97 40:134 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:26.856 INFO:tasks.workunit.client.0.smithi116.stdout:1.a 3 0 0 0 0 1449 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:20.387043+0000 32'97 41:137 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:42:20.386938+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T20:11:23.199988+0000 3 0 2023-11-02T08:42:26.856 INFO:tasks.workunit.client.0.smithi116.stdout:1.9 3 0 0 0 0 1111 0 0 14 84 14 stale+active+clean 2023-11-02T08:42:09.853448+0000 39'98 41:129 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:26.857 INFO:tasks.workunit.client.0.smithi116.stdout:1.8 3 0 0 0 0 1449 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:17.109406+0000 32'97 40:142 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:17.109289+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T08:57:51.194607+0000 3 0 2023-11-02T08:42:26.857 INFO:tasks.workunit.client.0.smithi116.stdout:1.7 3 0 0 0 0 1464 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:13.657784+0000 32'97 40:134 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:26.857 INFO:tasks.workunit.client.0.smithi116.stdout:1.6 4 0 0 0 0 1932 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:19.366709+0000 32'97 41:137 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:42:19.366599+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T11:09:38.248352+0000 4 0 2023-11-02T08:42:26.857 INFO:tasks.workunit.client.0.smithi116.stdout:1.5 3 0 0 0 0 1464 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:09.853054+0000 32'97 41:128 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:26.857 INFO:tasks.workunit.client.0.smithi116.stdout:1.4 2 0 0 0 0 966 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:18.171223+0000 32'97 40:142 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:18.171099+0000 32'97 2023-11-02T08:42:18.171099+0000 0 1 periodic scrub scheduled @ 2023-11-03T16:21:33.552952+0000 2 0 2023-11-02T08:42:26.858 INFO:tasks.workunit.client.0.smithi116.stdout:1.3 4 0 0 0 0 1948 0 0 14 84 14 stale+active+clean 2023-11-02T08:42:13.658020+0000 34'98 40:135 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:26.858 INFO:tasks.workunit.client.0.smithi116.stdout:1.2 1 0 0 0 0 483 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:12.415782+0000 32'97 40:137 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:42:12.415700+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T10:44:28.963015+0000 1 0 2023-11-02T08:42:26.858 INFO:tasks.workunit.client.0.smithi116.stdout:1.c 3 0 0 0 0 1464 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:06.229158+0000 32'97 39:130 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:26.858 INFO:tasks.workunit.client.0.smithi116.stdout:1.1 2 0 0 0 0 966 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:09.981554+0000 32'97 41:140 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:42:09.981464+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T13:24:58.648214+0000 2 0 2023-11-02T08:42:26.858 INFO:tasks.workunit.client.0.smithi116.stdout:1.1c 3 0 0 0 0 1464 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:06.229416+0000 32'97 39:130 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:26.859 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 3 0 0 0 0 1449 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:07.198418+0000 32'97 39:142 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:07.198292+0000 32'97 2023-11-02T08:42:07.198292+0000 0 1 periodic scrub scheduled @ 2023-11-03T16:56:03.107625+0000 3 0 2023-11-02T08:42:26.859 INFO:tasks.workunit.client.0.smithi116.stdout:1.d 3 0 0 0 0 1449 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:19.844826+0000 32'97 41:140 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:42:19.844682+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T12:02:33.908652+0000 3 0 2023-11-02T08:42:26.859 INFO:tasks.workunit.client.0.smithi116.stdout:1.e 3 0 0 0 0 981 0 0 15 84 15 stale+active+clean 2023-11-02T08:42:12.033008+0000 39'99 40:127 [0,1,2] 0 [0,1,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:26.859 INFO:tasks.workunit.client.0.smithi116.stdout:1.f 3 0 0 0 0 1464 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:13.657690+0000 32'97 40:134 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:26.859 INFO:tasks.workunit.client.0.smithi116.stdout:1.10 3 0 0 0 0 1464 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:06.230332+0000 32'97 39:130 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:26.860 INFO:tasks.workunit.client.0.smithi116.stdout:1.11 2 0 0 0 0 966 0 0 14 84 14 stale+active+clean 2023-11-02T08:42:18.867262+0000 39'98 41:141 [2,0,1] 2 [2,0,1] 2 39'98 2023-11-02T08:42:18.867158+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T09:47:05.817612+0000 2 0 2023-11-02T08:42:26.860 INFO:tasks.workunit.client.0.smithi116.stdout:1.12 1 0 0 0 0 483 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:15.453233+0000 32'97 40:137 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:42:15.453129+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T20:04:33.681148+0000 1 0 2023-11-02T08:42:26.860 INFO:tasks.workunit.client.0.smithi116.stdout:1.13 3 0 0 0 0 1464 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:13.582328+0000 32'97 40:134 [1,2,0] 1 [1,2,0] 1 32'97 2023-11-02T08:42:10.923269+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:26.860 INFO:tasks.workunit.client.0.smithi116.stdout:1.14 3 0 0 0 0 1464 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:06.228338+0000 32'97 39:130 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:26.860 INFO:tasks.workunit.client.0.smithi116.stdout:1.15 3 0 0 0 0 1449 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:14.857520+0000 32'97 41:140 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:42:14.857410+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T13:02:43.711974+0000 3 0 2023-11-02T08:42:26.861 INFO:tasks.workunit.client.0.smithi116.stdout:1.16 4 0 0 0 0 1932 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:16.428638+0000 32'97 40:137 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:42:16.428517+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T15:31:05.948447+0000 4 0 2023-11-02T08:42:26.861 INFO:tasks.workunit.client.0.smithi116.stdout:1.17 3 0 0 0 0 1464 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:13.658218+0000 32'97 40:134 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:26.861 INFO:tasks.workunit.client.0.smithi116.stdout:1.18 3 0 0 0 0 1464 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:06.228494+0000 32'97 39:130 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:26.861 INFO:tasks.workunit.client.0.smithi116.stdout:1.19 3 0 0 0 0 1464 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:09.853240+0000 32'97 41:128 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:26.861 INFO:tasks.workunit.client.0.smithi116.stdout:1.1a 3 0 0 0 0 1464 0 0 13 84 13 stale+active+clean 2023-11-02T08:42:12.033689+0000 32'97 40:125 [0,1,2] 0 [0,1,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:26.862 INFO:tasks.workunit.client.0.smithi116.stdout:1.1b 5 0 0 0 0 1594 0 0 16 84 16 stale+active+clean 2023-11-02T08:42:13.582327+0000 40'100 40:137 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:42:26.862 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:42:26.862 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:42:26.862 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:42:26.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2023-11-02T08:42:26.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 0 2023-11-02T08:42:26.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2023-11-02T08:42:26.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:42:26.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-11-02T08:42:26.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:42:26.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2023-11-02T08:42:26.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:42:26.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:42:26.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:42:26.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:42:26.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2023-11-02T08:42:26.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2023-11-02T08:42:26.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:42:26.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:42:26.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:42:26.872 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:42:26.872 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:42:26.872 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:42:26.872 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:42:26.872 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:42:26.873 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:42:26.873 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:42:26.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:42:26.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:42:26.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:42:26.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:42:26.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:42:26.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:42:26.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:42:26.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:42:26.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:42:26.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:42:26.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2023-11-02T08:42:26.875 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T08:42:26.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-11-02T08:42:26.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:42:26.877 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2023-11-02T08:42:26.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-11-02T08:42:26.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:42:26.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:42:26.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:42:26.901 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:42:26.898+0000 7f1943410500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:42:26.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:42:26.905+0000 7f1943410500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:42:26.909 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:42:26.906+0000 7f1943410500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:42:27.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-11-02T08:42:27.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:42:27.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:42:27.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:42:27.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:42:27.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:42:27.287 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:42:27.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:42:27.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:42:27.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:42:27.424 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:42:27.421+0000 7f1943410500 -1 Falling back to public interface 2023-11-02T08:42:27.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:42:28.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:42:28.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:42:28.723 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:42:28.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:42:28.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:42:28.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:42:29.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:42:30.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:42:30.114+0000 7f1943410500 -1 osd.0 38 log_to_monitors true 2023-11-02T08:42:30.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:42:30.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:42:30.192 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:42:30.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:42:30.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:42:30.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:42:30.633 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up out weight 0 up_from 54 up_thru 37 down_at 45 last_clean_interval [5,38) [v2:127.0.0.1:6802/120543,v1:127.0.0.1:6803/120543] [v2:127.0.0.1:6804/120543,v1:127.0.0.1:6805/120543] exists,up 584ee92b-5cfd-43f0-8535-9a70b0612348 2023-11-02T08:42:30.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:42:30.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:42:30.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:42:30.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2023-11-02T08:42:30.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 2 2023-11-02T08:42:30.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2023-11-02T08:42:30.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:42:30.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2023-11-02T08:42:30.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:42:30.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2023-11-02T08:42:30.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:42:30.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:42:30.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:42:30.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:42:30.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2023-11-02T08:42:30.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2023-11-02T08:42:30.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:42:30.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:42:30.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:42:30.641 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:42:30.641 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:42:30.641 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:42:30.641 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:42:30.641 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:42:30.642 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:42:30.642 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:42:30.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:42:30.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:42:30.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:42:30.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:42:30.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:42:30.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:42:30.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:42:30.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:42:30.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:42:30.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:42:30.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2023-11-02T08:42:30.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2023-11-02T08:42:30.645 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T08:42:30.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:42:30.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2023-11-02T08:42:30.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2023-11-02T08:42:30.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:42:30.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:42:30.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:42:30.668 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:42:30.664+0000 7f9653122500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:42:30.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:42:30.666+0000 7f9653122500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:42:30.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:42:30.667+0000 7f9653122500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:42:31.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2023-11-02T08:42:31.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:42:31.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T08:42:31.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:42:31.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:42:31.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:42:31.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:42:31.079 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:42:31.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:42:31.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:42:31.185 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:42:31.181+0000 7f9653122500 -1 Falling back to public interface 2023-11-02T08:42:31.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:42:32.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:42:32.503 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:42:32.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:42:32.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:42:32.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:42:32.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:42:32.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:42:33.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:42:33.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:42:33.930 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:42:33.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:42:33.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:42:33.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:42:34.268 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:42:34.264+0000 7f9653122500 -1 osd.2 39 log_to_monitors true 2023-11-02T08:42:34.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:42:35.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:42:35.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:42:35.336 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:42:35.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:42:35.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:42:35.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:42:35.717 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up out weight 0 up_from 58 up_thru 35 down_at 48 last_clean_interval [15,39) [v2:127.0.0.1:6810/121087,v1:127.0.0.1:6811/121087] [v2:127.0.0.1:6812/121087,v1:127.0.0.1:6813/121087] exists,up 8b4576fe-bdf3-4241-80eb-7d5e8439887c 2023-11-02T08:42:35.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:42:35.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:42:35.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:42:35.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2023-11-02T08:42:35.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.0 2023-11-02T08:42:36.304 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 is already in. 2023-11-02T08:42:36.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2023-11-02T08:42:36.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.2 2023-11-02T08:42:37.416 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 is already in. 2023-11-02T08:42:37.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2023-11-02T08:42:37.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: wait_for_clean 2023-11-02T08:42:37.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T08:42:37.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T08:42:37.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T08:42:37.429 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2023-11-02T08:42:37.430 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T08:42:37.430 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T08:42:37.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T08:42:37.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T08:42:37.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T08:42:37.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2023-11-02T08:42:37.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T08:42:37.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T08:42:37.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T08:42:37.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:42:37.481 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:42:38.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:42:38.001 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:42:38.002 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:42:38.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:42:38.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:42:38.003 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:42:38.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=231928233988 2023-11-02T08:42:38.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 231928233988 2023-11-02T08:42:38.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-231928233988' 2023-11-02T08:42:38.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:42:38.160 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:42:38.299 INFO:tasks.workunit.client.0.smithi116.stderr:Error ENXIO: problem getting command descriptions from osd.1 2023-11-02T08:42:38.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2023-11-02T08:42:38.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2023-11-02T08:42:38.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2023-11-02T08:42:38.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:42:38.302 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:42:38.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=249108103171 2023-11-02T08:42:38.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 249108103171 2023-11-02T08:42:38.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-231928233988 2-249108103171' 2023-11-02T08:42:38.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:42:38.423 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-231928233988 2023-11-02T08:42:38.424 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:42:38.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:42:38.424 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-231928233988 2023-11-02T08:42:38.425 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:42:38.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=231928233988 2023-11-02T08:42:38.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 231928233988' 2023-11-02T08:42:38.426 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 231928233988 2023-11-02T08:42:38.426 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:42:38.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 231928233986 -lt 231928233988 2023-11-02T08:42:38.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T08:42:39.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T08:42:39.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:42:40.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 231928233988 -lt 231928233988 2023-11-02T08:42:40.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:42:40.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-249108103171 2023-11-02T08:42:40.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:42:40.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:42:40.246 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-249108103171 2023-11-02T08:42:40.246 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:42:40.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=249108103171 2023-11-02T08:42:40.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 249108103171' 2023-11-02T08:42:40.247 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 249108103171 2023-11-02T08:42:40.248 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:42:40.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 249108103171 -lt 249108103171 2023-11-02T08:42:40.656 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T08:42:40.657 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:42:40.657 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:42:41.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2023-11-02T08:42:41.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:42:41.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:42:41.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:42:41.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:42:41.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:42:41.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:42:41.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:42:41.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:42:41.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:42:41.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:42:41.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:42:42.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2023-11-02T08:42:42.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2023-11-02T08:42:42.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2023-11-02T08:42:42.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2023-11-02T08:42:42.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:42:42.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T08:42:42.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:42:42.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:42:42.143 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:42:42.143 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:42:42.143 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:42:42.143 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:42:42.143 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:42:42.144 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:42:42.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:42:42.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:42:42.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:42:42.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:42:43.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2023-11-02T08:42:43.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:42:43.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:42:43.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:42:43.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:42:43.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:42:43.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:42:43.034 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:42:43.034 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:42:43.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:42:43.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:42:43.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2023-11-02T08:42:43.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:42:43.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2023-11-02T08:42:43.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:42:43.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:42:43.802 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:42:43.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:42:43.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:42:43.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:42:43.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:42:43.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:42:44.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:42:44.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:42:44.171 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:42:44.171 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:42:44.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2023-11-02T08:42:44.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:42:44.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:42:44.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:42:44.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:42:44.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:42:44.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:42:44.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:42:44.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:42:45.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:42:45.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:42:45.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2023-11-02T08:42:45.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:42:45.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2023-11-02T08:42:45.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:42:45.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:42:45.604 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:42:45.604 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:42:45.604 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:42:45.605 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:42:45.605 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:42:45.605 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:42:45.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:42:45.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:42:45.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:42:45.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:42:46.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2023-11-02T08:42:46.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:42:46.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:42:46.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:42:46.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:42:46.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:42:46.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:42:46.509 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:42:46.509 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:42:47.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:42:47.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:42:47.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2023-11-02T08:42:47.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:42:47.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2023-11-02T08:42:47.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:42:47.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:42:47.855 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:42:47.855 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:42:47.856 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:42:47.856 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:42:47.856 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:42:47.856 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:42:48.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:42:48.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:42:48.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:42:48.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:42:48.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2023-11-02T08:42:48.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:42:48.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:42:48.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:42:48.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:42:48.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:42:48.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:42:48.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:42:48.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:42:49.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:42:49.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:42:49.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2023-11-02T08:42:49.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:42:49.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2023-11-02T08:42:50.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:42:50.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:42:50.885 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:42:50.885 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:42:50.885 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:42:50.885 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:42:50.885 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:42:50.886 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:42:51.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:42:51.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:42:51.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:42:51.269 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:42:51.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2023-11-02T08:42:51.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:42:51.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:42:51.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:42:51.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:42:51.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:42:51.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:42:51.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:42:51.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:42:52.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:42:52.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:42:52.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2023-11-02T08:42:52.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:42:52.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2023-11-02T08:42:55.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:42:55.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:42:55.537 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:42:55.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:42:55.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:42:55.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:42:55.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:42:55.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:42:55.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:42:55.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:42:55.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:42:55.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:42:56.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2023-11-02T08:42:56.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:42:56.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:42:56.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:42:56.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:42:56.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:42:56.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:42:56.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:42:56.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:42:56.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:42:56.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:42:56.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2023-11-02T08:42:56.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:42:56.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2023-11-02T08:43:03.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:43:03.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:43:03.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:43:03.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:43:03.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:43:03.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:43:03.357 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:43:03.357 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:43:03.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:43:03.737 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:43:03.737 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:43:03.737 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:43:04.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2023-11-02T08:43:04.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:43:04.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:43:04.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:43:04.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:43:04.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:43:04.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:43:04.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:43:04.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:43:04.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:43:04.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:43:04.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2023-11-02T08:43:04.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:43:04.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2023-11-02T08:43:12.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:43:12.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:43:12.081 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:43:12.081 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:43:12.081 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:43:12.081 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:43:12.082 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:43:12.082 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:43:12.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:43:12.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:43:12.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:43:12.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:43:12.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2023-11-02T08:43:12.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:43:12.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:43:12.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:43:12.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:43:12.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:43:12.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:43:12.964 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:43:12.964 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:43:13.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:43:13.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:43:13.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2023-11-02T08:43:13.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2023-11-02T08:43:14.029 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T08:43:14.029 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_fingerprint": "9265913a-476f-4246-8d00-7318d249d9e9", 2023-11-02T08:43:14.030 INFO:tasks.workunit.client.0.smithi116.stdout: "version": "18.2.0-1159-gb5a9486b", 2023-11-02T08:43:14.030 INFO:tasks.workunit.client.0.smithi116.stdout: "commit": "b5a9486b4a6f144f83c51b7ec33111a2c86dc081", 2023-11-02T08:43:14.030 INFO:tasks.workunit.client.0.smithi116.stdout: "timestamp": "2023-11-02T08:43:14.025750+0000", 2023-11-02T08:43:14.030 INFO:tasks.workunit.client.0.smithi116.stdout: "tag": "", 2023-11-02T08:43:14.030 INFO:tasks.workunit.client.0.smithi116.stdout: "health": { 2023-11-02T08:43:14.031 INFO:tasks.workunit.client.0.smithi116.stdout: "status": "HEALTH_WARN", 2023-11-02T08:43:14.031 INFO:tasks.workunit.client.0.smithi116.stdout: "checks": { 2023-11-02T08:43:14.031 INFO:tasks.workunit.client.0.smithi116.stdout: "OSDMAP_FLAGS": { 2023-11-02T08:43:14.031 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:43:14.031 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:43:14.032 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "nodown,noout,noin flag(s) set", 2023-11-02T08:43:14.032 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 17 2023-11-02T08:43:14.032 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.032 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [], 2023-11-02T08:43:14.032 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:43:14.032 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.033 INFO:tasks.workunit.client.0.smithi116.stdout: "OSD_DOWN": { 2023-11-02T08:43:14.033 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:43:14.033 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:43:14.033 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "1 osds down", 2023-11-02T08:43:14.033 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 1 2023-11-02T08:43:14.034 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.034 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:43:14.034 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.035 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "osd.1 (root=default,host=smithi116) is down" 2023-11-02T08:43:14.035 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.035 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:14.035 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:43:14.035 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.036 INFO:tasks.workunit.client.0.smithi116.stdout: "PG_DEGRADED": { 2023-11-02T08:43:14.036 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:43:14.036 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:43:14.036 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "Degraded data redundancy: 96/288 objects degraded (33.333%), 32 pgs degraded", 2023-11-02T08:43:14.036 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 32 2023-11-02T08:43:14.036 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.037 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:43:14.037 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.037 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2023-11-02T08:43:14.037 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.037 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.038 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [2,0]" 2023-11-02T08:43:14.038 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.038 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.038 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2023-11-02T08:43:14.038 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.039 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.039 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [2,0]" 2023-11-02T08:43:14.039 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.039 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.039 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2023-11-02T08:43:14.040 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.040 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.040 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [2,0]" 2023-11-02T08:43:14.040 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.040 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.040 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2023-11-02T08:43:14.041 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.041 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.041 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [2,0]" 2023-11-02T08:43:14.041 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.041 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.042 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [2,0]" 2023-11-02T08:43:14.042 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.042 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.042 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [2,0]" 2023-11-02T08:43:14.042 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.042 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.043 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.a is active+undersized+degraded, acting [2,0]" 2023-11-02T08:43:14.043 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.043 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.043 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2023-11-02T08:43:14.043 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.044 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.044 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2023-11-02T08:43:14.044 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.045 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.045 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.d is active+undersized+degraded, acting [2,0]" 2023-11-02T08:43:14.045 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.045 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.045 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.e is active+undersized+degraded, acting [0,2]" 2023-11-02T08:43:14.046 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.046 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.046 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.f is active+undersized+degraded, acting [2,0]" 2023-11-02T08:43:14.046 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.046 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.046 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2023-11-02T08:43:14.047 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.047 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.047 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [2,0]" 2023-11-02T08:43:14.047 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.047 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.048 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2023-11-02T08:43:14.048 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.048 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.048 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [2,0]" 2023-11-02T08:43:14.048 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.048 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.049 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [0,2]" 2023-11-02T08:43:14.049 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.049 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.049 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [2,0]" 2023-11-02T08:43:14.049 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.050 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.050 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2023-11-02T08:43:14.050 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.050 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.050 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [2,0]" 2023-11-02T08:43:14.051 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.051 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.051 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [2,0]" 2023-11-02T08:43:14.051 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.051 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.052 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [2,0]" 2023-11-02T08:43:14.052 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.052 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.052 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [2,0]" 2023-11-02T08:43:14.052 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.052 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.053 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2023-11-02T08:43:14.053 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.053 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.053 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2023-11-02T08:43:14.053 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.054 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.054 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [2,0]" 2023-11-02T08:43:14.054 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.054 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.054 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [0,2]" 2023-11-02T08:43:14.054 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.055 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.055 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [2,0]" 2023-11-02T08:43:14.056 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.056 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:14.056 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:43:14.056 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.056 INFO:tasks.workunit.client.0.smithi116.stdout: "POOL_APP_NOT_ENABLED": { 2023-11-02T08:43:14.056 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:43:14.057 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:43:14.057 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "1 pool(s) do not have an application enabled", 2023-11-02T08:43:14.057 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 1 2023-11-02T08:43:14.057 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.057 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:43:14.058 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.058 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "application not enabled on pool 'test'" 2023-11-02T08:43:14.058 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.058 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.058 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2023-11-02T08:43:14.059 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.059 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:14.059 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:43:14.059 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.059 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.059 INFO:tasks.workunit.client.0.smithi116.stdout: "mutes": [] 2023-11-02T08:43:14.060 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.060 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap_first_committed": 1, 2023-11-02T08:43:14.060 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap_last_committed": 1, 2023-11-02T08:43:14.060 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap": { 2023-11-02T08:43:14.060 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 1, 2023-11-02T08:43:14.061 INFO:tasks.workunit.client.0.smithi116.stdout: "fsid": "0c9a372f-cec7-47d9-a309-8248453aabeb", 2023-11-02T08:43:14.061 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:39:57.418791Z", 2023-11-02T08:43:14.061 INFO:tasks.workunit.client.0.smithi116.stdout: "created": "2023-11-02T08:39:57.418791Z", 2023-11-02T08:43:14.061 INFO:tasks.workunit.client.0.smithi116.stdout: "min_mon_release": 18, 2023-11-02T08:43:14.062 INFO:tasks.workunit.client.0.smithi116.stdout: "min_mon_release_name": "reef", 2023-11-02T08:43:14.062 INFO:tasks.workunit.client.0.smithi116.stdout: "election_strategy": 1, 2023-11-02T08:43:14.062 INFO:tasks.workunit.client.0.smithi116.stdout: "disallowed_leaders: ": "", 2023-11-02T08:43:14.063 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode": false, 2023-11-02T08:43:14.063 INFO:tasks.workunit.client.0.smithi116.stdout: "tiebreaker_mon": "", 2023-11-02T08:43:14.063 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_ranks: ": "", 2023-11-02T08:43:14.063 INFO:tasks.workunit.client.0.smithi116.stdout: "features": { 2023-11-02T08:43:14.063 INFO:tasks.workunit.client.0.smithi116.stdout: "persistent": [ 2023-11-02T08:43:14.064 INFO:tasks.workunit.client.0.smithi116.stdout: "kraken", 2023-11-02T08:43:14.064 INFO:tasks.workunit.client.0.smithi116.stdout: "luminous", 2023-11-02T08:43:14.064 INFO:tasks.workunit.client.0.smithi116.stdout: "mimic", 2023-11-02T08:43:14.064 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap-prune", 2023-11-02T08:43:14.064 INFO:tasks.workunit.client.0.smithi116.stdout: "nautilus", 2023-11-02T08:43:14.064 INFO:tasks.workunit.client.0.smithi116.stdout: "octopus", 2023-11-02T08:43:14.065 INFO:tasks.workunit.client.0.smithi116.stdout: "pacific", 2023-11-02T08:43:14.065 INFO:tasks.workunit.client.0.smithi116.stdout: "elector-pinging", 2023-11-02T08:43:14.065 INFO:tasks.workunit.client.0.smithi116.stdout: "quincy", 2023-11-02T08:43:14.065 INFO:tasks.workunit.client.0.smithi116.stdout: "reef" 2023-11-02T08:43:14.065 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:14.066 INFO:tasks.workunit.client.0.smithi116.stdout: "optional": [] 2023-11-02T08:43:14.066 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.066 INFO:tasks.workunit.client.0.smithi116.stdout: "mons": [ 2023-11-02T08:43:14.066 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.066 INFO:tasks.workunit.client.0.smithi116.stdout: "rank": 0, 2023-11-02T08:43:14.067 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "a", 2023-11-02T08:43:14.067 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:43:14.067 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:14.067 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.067 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:14.067 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:7115", 2023-11-02T08:43:14.068 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 0 2023-11-02T08:43:14.068 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.068 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.068 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.068 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:7115/0", 2023-11-02T08:43:14.069 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:7115/0", 2023-11-02T08:43:14.069 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 0, 2023-11-02T08:43:14.069 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 0, 2023-11-02T08:43:14.069 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_location": "{}" 2023-11-02T08:43:14.069 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.069 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.070 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.070 INFO:tasks.workunit.client.0.smithi116.stdout: "quorum": [ 2023-11-02T08:43:14.070 INFO:tasks.workunit.client.0.smithi116.stdout: 0 2023-11-02T08:43:14.070 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:14.070 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap": { 2023-11-02T08:43:14.071 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 66, 2023-11-02T08:43:14.071 INFO:tasks.workunit.client.0.smithi116.stdout: "fsid": "0c9a372f-cec7-47d9-a309-8248453aabeb", 2023-11-02T08:43:14.071 INFO:tasks.workunit.client.0.smithi116.stdout: "created": "2023-11-02T08:39:58.207786+0000", 2023-11-02T08:43:14.071 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:42:37.824722+0000", 2023-11-02T08:43:14.071 INFO:tasks.workunit.client.0.smithi116.stdout: "last_up_change": "2023-11-02T08:42:34.544550+0000", 2023-11-02T08:43:14.071 INFO:tasks.workunit.client.0.smithi116.stdout: "last_in_change": "2023-11-02T08:42:36.775521+0000", 2023-11-02T08:43:14.072 INFO:tasks.workunit.client.0.smithi116.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-11-02T08:43:14.072 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_num": 5800384, 2023-11-02T08:43:14.072 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_set": [ 2023-11-02T08:43:14.072 INFO:tasks.workunit.client.0.smithi116.stdout: "nodown", 2023-11-02T08:43:14.072 INFO:tasks.workunit.client.0.smithi116.stdout: "noin", 2023-11-02T08:43:14.073 INFO:tasks.workunit.client.0.smithi116.stdout: "noout", 2023-11-02T08:43:14.073 INFO:tasks.workunit.client.0.smithi116.stdout: "pglog_hardlimit", 2023-11-02T08:43:14.073 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snapdirs", 2023-11-02T08:43:14.073 INFO:tasks.workunit.client.0.smithi116.stdout: "recovery_deletes", 2023-11-02T08:43:14.073 INFO:tasks.workunit.client.0.smithi116.stdout: "sortbitwise" 2023-11-02T08:43:14.074 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:14.074 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_version": 9, 2023-11-02T08:43:14.074 INFO:tasks.workunit.client.0.smithi116.stdout: "full_ratio": 0.99000000953674316, 2023-11-02T08:43:14.074 INFO:tasks.workunit.client.0.smithi116.stdout: "backfillfull_ratio": 0.99000000953674316, 2023-11-02T08:43:14.074 INFO:tasks.workunit.client.0.smithi116.stdout: "nearfull_ratio": 0.85000002384185791, 2023-11-02T08:43:14.075 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_snapshot": "", 2023-11-02T08:43:14.075 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_max": 1, 2023-11-02T08:43:14.075 INFO:tasks.workunit.client.0.smithi116.stdout: "max_osd": 3, 2023-11-02T08:43:14.075 INFO:tasks.workunit.client.0.smithi116.stdout: "require_min_compat_client": "luminous", 2023-11-02T08:43:14.075 INFO:tasks.workunit.client.0.smithi116.stdout: "min_compat_client": "jewel", 2023-11-02T08:43:14.075 INFO:tasks.workunit.client.0.smithi116.stdout: "require_osd_release": "reef", 2023-11-02T08:43:14.076 INFO:tasks.workunit.client.0.smithi116.stdout: "allow_crimson": false, 2023-11-02T08:43:14.076 INFO:tasks.workunit.client.0.smithi116.stdout: "pools": [ 2023-11-02T08:43:14.076 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.076 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": 1, 2023-11-02T08:43:14.076 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_name": "test", 2023-11-02T08:43:14.077 INFO:tasks.workunit.client.0.smithi116.stdout: "create_time": "2023-11-02T08:41:38.449621+0000", 2023-11-02T08:43:14.077 INFO:tasks.workunit.client.0.smithi116.stdout: "flags": 1, 2023-11-02T08:43:14.077 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_names": "hashpspool", 2023-11-02T08:43:14.077 INFO:tasks.workunit.client.0.smithi116.stdout: "type": 1, 2023-11-02T08:43:14.077 INFO:tasks.workunit.client.0.smithi116.stdout: "size": 3, 2023-11-02T08:43:14.077 INFO:tasks.workunit.client.0.smithi116.stdout: "min_size": 2, 2023-11-02T08:43:14.078 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_rule": 0, 2023-11-02T08:43:14.078 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_count": 0, 2023-11-02T08:43:14.078 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_target": 0, 2023-11-02T08:43:14.078 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_barrier": 0, 2023-11-02T08:43:14.078 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-11-02T08:43:14.079 INFO:tasks.workunit.client.0.smithi116.stdout: "object_hash": 2, 2023-11-02T08:43:14.079 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_autoscale_mode": "on", 2023-11-02T08:43:14.079 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num": 32, 2023-11-02T08:43:14.079 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_placement_num": 14, 2023-11-02T08:43:14.079 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_placement_num_target": 32, 2023-11-02T08:43:14.080 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num_target": 32, 2023-11-02T08:43:14.080 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num_pending": 32, 2023-11-02T08:43:14.080 INFO:tasks.workunit.client.0.smithi116.stdout: "last_pg_merge_meta": { 2023-11-02T08:43:14.080 INFO:tasks.workunit.client.0.smithi116.stdout: "source_pgid": "0.0", 2023-11-02T08:43:14.080 INFO:tasks.workunit.client.0.smithi116.stdout: "ready_epoch": 0, 2023-11-02T08:43:14.080 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_started": 0, 2023-11-02T08:43:14.081 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_clean": 0, 2023-11-02T08:43:14.081 INFO:tasks.workunit.client.0.smithi116.stdout: "source_version": "0'0", 2023-11-02T08:43:14.081 INFO:tasks.workunit.client.0.smithi116.stdout: "target_version": "0'0" 2023-11-02T08:43:14.081 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.081 INFO:tasks.workunit.client.0.smithi116.stdout: "last_change": "64", 2023-11-02T08:43:14.081 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend": "0", 2023-11-02T08:43:14.082 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend_prenautilus": "0", 2023-11-02T08:43:14.082 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend_preluminous": "33", 2023-11-02T08:43:14.082 INFO:tasks.workunit.client.0.smithi116.stdout: "auid": 0, 2023-11-02T08:43:14.082 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_mode": "selfmanaged", 2023-11-02T08:43:14.082 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_seq": 0, 2023-11-02T08:43:14.083 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_epoch": 0, 2023-11-02T08:43:14.083 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_snaps": [], 2023-11-02T08:43:14.083 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_snaps": "[]", 2023-11-02T08:43:14.083 INFO:tasks.workunit.client.0.smithi116.stdout: "quota_max_bytes": 0, 2023-11-02T08:43:14.083 INFO:tasks.workunit.client.0.smithi116.stdout: "quota_max_objects": 0, 2023-11-02T08:43:14.083 INFO:tasks.workunit.client.0.smithi116.stdout: "tiers": [], 2023-11-02T08:43:14.084 INFO:tasks.workunit.client.0.smithi116.stdout: "tier_of": -1, 2023-11-02T08:43:14.084 INFO:tasks.workunit.client.0.smithi116.stdout: "read_tier": -1, 2023-11-02T08:43:14.084 INFO:tasks.workunit.client.0.smithi116.stdout: "write_tier": -1, 2023-11-02T08:43:14.084 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_mode": "none", 2023-11-02T08:43:14.084 INFO:tasks.workunit.client.0.smithi116.stdout: "target_max_bytes": 0, 2023-11-02T08:43:14.085 INFO:tasks.workunit.client.0.smithi116.stdout: "target_max_objects": 0, 2023-11-02T08:43:14.085 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-11-02T08:43:14.085 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-11-02T08:43:14.085 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_full_ratio_micro": 800000, 2023-11-02T08:43:14.085 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_min_flush_age": 0, 2023-11-02T08:43:14.085 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_min_evict_age": 0, 2023-11-02T08:43:14.086 INFO:tasks.workunit.client.0.smithi116.stdout: "erasure_code_profile": "", 2023-11-02T08:43:14.086 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_params": { 2023-11-02T08:43:14.086 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "none" 2023-11-02T08:43:14.086 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.086 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_period": 0, 2023-11-02T08:43:14.087 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_count": 0, 2023-11-02T08:43:14.087 INFO:tasks.workunit.client.0.smithi116.stdout: "use_gmt_hitset": true, 2023-11-02T08:43:14.087 INFO:tasks.workunit.client.0.smithi116.stdout: "min_read_recency_for_promote": 0, 2023-11-02T08:43:14.087 INFO:tasks.workunit.client.0.smithi116.stdout: "min_write_recency_for_promote": 0, 2023-11-02T08:43:14.087 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_grade_decay_rate": 0, 2023-11-02T08:43:14.087 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_search_last_n": 0, 2023-11-02T08:43:14.088 INFO:tasks.workunit.client.0.smithi116.stdout: "grade_table": [], 2023-11-02T08:43:14.088 INFO:tasks.workunit.client.0.smithi116.stdout: "stripe_width": 0, 2023-11-02T08:43:14.088 INFO:tasks.workunit.client.0.smithi116.stdout: "expected_num_objects": 0, 2023-11-02T08:43:14.088 INFO:tasks.workunit.client.0.smithi116.stdout: "fast_read": false, 2023-11-02T08:43:14.088 INFO:tasks.workunit.client.0.smithi116.stdout: "options": {}, 2023-11-02T08:43:14.089 INFO:tasks.workunit.client.0.smithi116.stdout: "application_metadata": {}, 2023-11-02T08:43:14.089 INFO:tasks.workunit.client.0.smithi116.stdout: "read_balance": { 2023-11-02T08:43:14.089 INFO:tasks.workunit.client.0.smithi116.stdout: "score_acting": 1.690000057220459, 2023-11-02T08:43:14.089 INFO:tasks.workunit.client.0.smithi116.stdout: "score_stable": 1.690000057220459, 2023-11-02T08:43:14.089 INFO:tasks.workunit.client.0.smithi116.stdout: "optimal_score": 0.67000001668930054, 2023-11-02T08:43:14.089 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_score_acting": 1.1299999952316284, 2023-11-02T08:43:14.090 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_score_stable": 1.1299999952316284, 2023-11-02T08:43:14.090 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity_weighted": 1, 2023-11-02T08:43:14.090 INFO:tasks.workunit.client.0.smithi116.stdout: "average_primary_affinity": 1, 2023-11-02T08:43:14.090 INFO:tasks.workunit.client.0.smithi116.stdout: "average_primary_affinity_weighted": 1 2023-11-02T08:43:14.090 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.091 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.091 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:14.091 INFO:tasks.workunit.client.0.smithi116.stdout: "osds": [ 2023-11-02T08:43:14.091 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.091 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:43:14.091 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "584ee92b-5cfd-43f0-8535-9a70b0612348", 2023-11-02T08:43:14.092 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 1, 2023-11-02T08:43:14.092 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:43:14.092 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:43:14.092 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:43:14.092 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 5, 2023-11-02T08:43:14.093 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 38, 2023-11-02T08:43:14.093 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 54, 2023-11-02T08:43:14.093 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 63, 2023-11-02T08:43:14.093 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 45, 2023-11-02T08:43:14.093 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:43:14.093 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:43:14.094 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:14.094 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.094 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:14.094 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6802", 2023-11-02T08:43:14.094 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 120543 2023-11-02T08:43:14.095 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.095 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.095 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:14.095 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6803", 2023-11-02T08:43:14.095 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 120543 2023-11-02T08:43:14.096 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.096 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.096 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.096 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:43:14.096 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:14.096 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.097 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:14.097 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6804", 2023-11-02T08:43:14.097 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 120543 2023-11-02T08:43:14.097 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.097 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.098 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:14.098 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6805", 2023-11-02T08:43:14.098 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 120543 2023-11-02T08:43:14.098 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.098 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.098 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.099 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:43:14.099 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:14.099 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.099 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:14.099 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6808", 2023-11-02T08:43:14.100 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 120543 2023-11-02T08:43:14.100 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.100 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.100 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:14.100 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6809", 2023-11-02T08:43:14.100 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 120543 2023-11-02T08:43:14.101 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.101 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.101 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.101 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:43:14.101 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:14.102 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.102 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:14.102 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6806", 2023-11-02T08:43:14.102 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 120543 2023-11-02T08:43:14.102 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.102 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.103 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:14.103 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6807", 2023-11-02T08:43:14.103 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 120543 2023-11-02T08:43:14.103 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.103 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.104 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.104 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6803/120543", 2023-11-02T08:43:14.104 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6805/120543", 2023-11-02T08:43:14.104 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6809/120543", 2023-11-02T08:43:14.104 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6807/120543", 2023-11-02T08:43:14.104 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:43:14.105 INFO:tasks.workunit.client.0.smithi116.stdout: "exists", 2023-11-02T08:43:14.105 INFO:tasks.workunit.client.0.smithi116.stdout: "up" 2023-11-02T08:43:14.105 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.105 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.105 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.106 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:43:14.106 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "4d681ad5-42d6-443f-ac67-86739e3ca916", 2023-11-02T08:43:14.106 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 0, 2023-11-02T08:43:14.106 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:43:14.106 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:43:14.106 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:43:14.107 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 0, 2023-11-02T08:43:14.107 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 0, 2023-11-02T08:43:14.107 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 10, 2023-11-02T08:43:14.107 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 43, 2023-11-02T08:43:14.107 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 46, 2023-11-02T08:43:14.108 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:43:14.108 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:43:14.108 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:14.108 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.108 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:14.108 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6810", 2023-11-02T08:43:14.109 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 114539 2023-11-02T08:43:14.109 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.109 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.109 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:14.109 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6811", 2023-11-02T08:43:14.110 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 114539 2023-11-02T08:43:14.110 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.110 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.110 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.110 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:43:14.110 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:14.111 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.111 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:14.111 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6812", 2023-11-02T08:43:14.111 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 114539 2023-11-02T08:43:14.111 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.112 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.112 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:14.112 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6813", 2023-11-02T08:43:14.112 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 114539 2023-11-02T08:43:14.112 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.112 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.113 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.113 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:43:14.113 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:14.113 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.113 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:14.114 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6816", 2023-11-02T08:43:14.114 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 114539 2023-11-02T08:43:14.114 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.114 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.114 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:14.114 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6817", 2023-11-02T08:43:14.115 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 114539 2023-11-02T08:43:14.115 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.115 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.115 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.115 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:43:14.116 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:14.116 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.116 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:14.116 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6814", 2023-11-02T08:43:14.116 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 114539 2023-11-02T08:43:14.117 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.117 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.117 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:14.117 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6815", 2023-11-02T08:43:14.117 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 114539 2023-11-02T08:43:14.117 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.118 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.118 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.118 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6811/114539", 2023-11-02T08:43:14.118 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6813/114539", 2023-11-02T08:43:14.118 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6817/114539", 2023-11-02T08:43:14.119 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6815/114539", 2023-11-02T08:43:14.119 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:43:14.119 INFO:tasks.workunit.client.0.smithi116.stdout: "exists" 2023-11-02T08:43:14.119 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.119 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.119 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.120 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:43:14.120 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "8b4576fe-bdf3-4241-80eb-7d5e8439887c", 2023-11-02T08:43:14.120 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 1, 2023-11-02T08:43:14.120 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:43:14.120 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:43:14.121 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:43:14.121 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 15, 2023-11-02T08:43:14.121 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 39, 2023-11-02T08:43:14.121 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 58, 2023-11-02T08:43:14.122 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 65, 2023-11-02T08:43:14.122 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 48, 2023-11-02T08:43:14.122 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:43:14.122 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:43:14.123 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:14.123 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.123 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:14.123 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6810", 2023-11-02T08:43:14.123 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 121087 2023-11-02T08:43:14.124 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.124 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.124 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:14.124 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6811", 2023-11-02T08:43:14.124 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 121087 2023-11-02T08:43:14.124 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.125 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.125 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.125 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:43:14.125 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:14.126 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.126 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:14.126 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6812", 2023-11-02T08:43:14.126 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 121087 2023-11-02T08:43:14.127 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.127 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.127 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:14.127 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6813", 2023-11-02T08:43:14.127 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 121087 2023-11-02T08:43:14.128 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.128 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.128 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.128 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:43:14.128 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:14.128 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.129 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:14.129 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6816", 2023-11-02T08:43:14.129 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 121087 2023-11-02T08:43:14.129 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.129 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.130 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:14.130 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6817", 2023-11-02T08:43:14.130 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 121087 2023-11-02T08:43:14.130 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.130 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.130 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.131 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:43:14.131 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:14.131 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.131 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:14.131 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6814", 2023-11-02T08:43:14.132 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 121087 2023-11-02T08:43:14.132 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.132 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.132 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:14.132 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6815", 2023-11-02T08:43:14.132 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 121087 2023-11-02T08:43:14.133 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.133 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.133 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.133 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6811/121087", 2023-11-02T08:43:14.133 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6813/121087", 2023-11-02T08:43:14.134 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6817/121087", 2023-11-02T08:43:14.134 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6815/121087", 2023-11-02T08:43:14.134 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:43:14.134 INFO:tasks.workunit.client.0.smithi116.stdout: "exists", 2023-11-02T08:43:14.134 INFO:tasks.workunit.client.0.smithi116.stdout: "up" 2023-11-02T08:43:14.135 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.135 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.135 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:14.135 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_xinfo": [ 2023-11-02T08:43:14.135 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.135 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:43:14.136 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:42:23.925605+0000", 2023-11-02T08:43:14.136 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:43:14.136 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:43:14.136 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:43:14.136 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:43:14.136 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:40:28.898043+0000", 2023-11-02T08:43:14.137 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:43:14.137 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.137 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.137 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:43:14.137 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:42:24.478710+0000", 2023-11-02T08:43:14.138 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:43:14.138 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:43:14.138 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:43:14.138 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:43:14.138 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:41:02.215113+0000", 2023-11-02T08:43:14.139 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:43:14.139 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.139 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.139 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:43:14.139 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:42:25.101538+0000", 2023-11-02T08:43:14.139 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:43:14.140 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:43:14.140 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:43:14.140 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:43:14.140 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:41:35.175518+0000", 2023-11-02T08:43:14.140 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:43:14.141 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.141 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:14.141 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap": [], 2023-11-02T08:43:14.141 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap_items": [], 2023-11-02T08:43:14.141 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap_primaries": [], 2023-11-02T08:43:14.141 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_temp": [], 2023-11-02T08:43:14.142 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_temp": [], 2023-11-02T08:43:14.142 INFO:tasks.workunit.client.0.smithi116.stdout: "blocklist": {}, 2023-11-02T08:43:14.142 INFO:tasks.workunit.client.0.smithi116.stdout: "range_blocklist": {}, 2023-11-02T08:43:14.142 INFO:tasks.workunit.client.0.smithi116.stdout: "erasure_code_profiles": { 2023-11-02T08:43:14.142 INFO:tasks.workunit.client.0.smithi116.stdout: "default": { 2023-11-02T08:43:14.143 INFO:tasks.workunit.client.0.smithi116.stdout: "k": "2", 2023-11-02T08:43:14.143 INFO:tasks.workunit.client.0.smithi116.stdout: "m": "2", 2023-11-02T08:43:14.143 INFO:tasks.workunit.client.0.smithi116.stdout: "plugin": "jerasure", 2023-11-02T08:43:14.143 INFO:tasks.workunit.client.0.smithi116.stdout: "technique": "reed_sol_van" 2023-11-02T08:43:14.143 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.143 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.144 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_snaps_queue": [], 2023-11-02T08:43:14.144 INFO:tasks.workunit.client.0.smithi116.stdout: "new_removed_snaps": [], 2023-11-02T08:43:14.144 INFO:tasks.workunit.client.0.smithi116.stdout: "new_purged_snaps": [], 2023-11-02T08:43:14.144 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_node_flags": {}, 2023-11-02T08:43:14.144 INFO:tasks.workunit.client.0.smithi116.stdout: "device_class_flags": {}, 2023-11-02T08:43:14.145 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode": { 2023-11-02T08:43:14.145 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode_enabled": false, 2023-11-02T08:43:14.145 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_bucket_count": 0, 2023-11-02T08:43:14.145 INFO:tasks.workunit.client.0.smithi116.stdout: "degraded_stretch_mode": 0, 2023-11-02T08:43:14.145 INFO:tasks.workunit.client.0.smithi116.stdout: "recovering_stretch_mode": 0, 2023-11-02T08:43:14.145 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode_bucket": 0 2023-11-02T08:43:14.146 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.146 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.146 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_metadata": [ 2023-11-02T08:43:14.146 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.146 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:43:14.147 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:43:14.147 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6804/120543,v1:127.0.0.1:6805/120543]", 2023-11-02T08:43:14.147 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:43:14.147 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:43:14.147 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:43:14.147 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:43:14.148 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:43:14.148 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:43:14.148 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:43:14.148 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:43:14.148 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:43:14.149 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:43:14.149 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2023-11-02T08:43:14.149 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:43:14.149 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:43:14.149 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:43:14.149 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:43:14.150 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:43:14.150 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:43:14.150 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:43:14.150 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:43:14.150 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:43:14.151 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:43:14.151 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:40:04.089233Z", 2023-11-02T08:43:14.151 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:43:14.151 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:43:14.151 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:43:14.151 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:43:14.152 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:43:14.152 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:43:14.152 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:43:14.152 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6802/120543,v1:127.0.0.1:6803/120543]", 2023-11-02T08:43:14.152 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:43:14.153 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/120543,v1:127.0.0.1:6809/120543]", 2023-11-02T08:43:14.153 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/120543,v1:127.0.0.1:6807/120543]", 2023-11-02T08:43:14.153 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:43:14.153 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:43:14.153 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:43:14.153 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:43:14.154 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:43:14.154 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:43:14.154 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:43:14.154 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:43:14.154 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:43:14.155 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/0", 2023-11-02T08:43:14.155 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:43:14.155 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:43:14.155 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:43:14.155 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.155 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.156 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:43:14.156 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:43:14.156 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6812/114539,v1:127.0.0.1:6813/114539]", 2023-11-02T08:43:14.156 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:43:14.156 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:43:14.157 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:43:14.157 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:43:14.157 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:43:14.157 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:43:14.157 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:43:14.157 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:43:14.158 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:43:14.158 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:43:14.158 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2023-11-02T08:43:14.158 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:43:14.158 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:43:14.159 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:43:14.159 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:43:14.159 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:43:14.159 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:43:14.159 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:43:14.160 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:43:14.160 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:43:14.160 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:43:14.160 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:40:36.033676Z", 2023-11-02T08:43:14.160 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:43:14.160 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:43:14.161 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:43:14.161 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:43:14.161 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:43:14.161 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:43:14.161 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:43:14.161 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6810/114539,v1:127.0.0.1:6811/114539]", 2023-11-02T08:43:14.162 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:43:14.162 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/114539,v1:127.0.0.1:6817/114539]", 2023-11-02T08:43:14.162 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/114539,v1:127.0.0.1:6815/114539]", 2023-11-02T08:43:14.162 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:43:14.162 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:43:14.163 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:43:14.163 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:43:14.163 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:43:14.163 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:43:14.163 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:43:14.164 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:43:14.164 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:43:14.164 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/1", 2023-11-02T08:43:14.164 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:43:14.164 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:43:14.164 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:43:14.165 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.165 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.165 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:43:14.165 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:43:14.165 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6812/121087,v1:127.0.0.1:6813/121087]", 2023-11-02T08:43:14.166 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:43:14.166 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:43:14.166 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:43:14.166 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:43:14.166 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:43:14.166 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:43:14.167 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:43:14.167 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:43:14.167 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:43:14.167 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:43:14.167 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2023-11-02T08:43:14.168 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:43:14.168 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:43:14.168 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:43:14.168 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:43:14.168 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:43:14.168 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:43:14.169 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:43:14.169 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:43:14.169 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:43:14.169 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:43:14.169 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:41:08.750575Z", 2023-11-02T08:43:14.170 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:43:14.170 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:43:14.170 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:43:14.170 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:43:14.170 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:43:14.170 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:43:14.171 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:43:14.171 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6810/121087,v1:127.0.0.1:6811/121087]", 2023-11-02T08:43:14.171 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:43:14.171 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/121087,v1:127.0.0.1:6817/121087]", 2023-11-02T08:43:14.171 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/121087,v1:127.0.0.1:6815/121087]", 2023-11-02T08:43:14.172 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:43:14.172 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:43:14.172 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:43:14.172 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:43:14.172 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:43:14.172 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:43:14.173 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:43:14.173 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:43:14.173 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:43:14.173 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/2", 2023-11-02T08:43:14.173 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:43:14.174 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:43:14.174 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:43:14.174 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.174 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:14.174 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_clean_epochs": { 2023-11-02T08:43:14.174 INFO:tasks.workunit.client.0.smithi116.stdout: "min_last_epoch_clean": 0, 2023-11-02T08:43:14.175 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_clean": { 2023-11-02T08:43:14.175 INFO:tasks.workunit.client.0.smithi116.stdout: "per_pool": [ 2023-11-02T08:43:14.175 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.175 INFO:tasks.workunit.client.0.smithi116.stdout: "poolid": 1, 2023-11-02T08:43:14.175 INFO:tasks.workunit.client.0.smithi116.stdout: "floor": 36 2023-11-02T08:43:14.176 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.176 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.176 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.176 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_epochs": [ 2023-11-02T08:43:14.176 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.176 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:43:14.177 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 55 2023-11-02T08:43:14.177 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.177 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.177 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:43:14.177 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 10 2023-11-02T08:43:14.178 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.178 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.178 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:43:14.178 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 59 2023-11-02T08:43:14.178 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.178 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.179 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.179 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_first_committed": 1, 2023-11-02T08:43:14.179 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_last_committed": 66, 2023-11-02T08:43:14.179 INFO:tasks.workunit.client.0.smithi116.stdout: "crushmap": { 2023-11-02T08:43:14.179 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": [ 2023-11-02T08:43:14.180 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.180 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:43:14.180 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.0", 2023-11-02T08:43:14.180 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:43:14.180 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.180 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.181 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:43:14.181 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.1", 2023-11-02T08:43:14.181 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:43:14.181 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.181 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.182 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:43:14.182 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.2", 2023-11-02T08:43:14.182 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:43:14.182 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.182 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:14.182 INFO:tasks.workunit.client.0.smithi116.stdout: "types": [ 2023-11-02T08:43:14.183 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.183 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 0, 2023-11-02T08:43:14.183 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd" 2023-11-02T08:43:14.183 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.183 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.184 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:43:14.184 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "host" 2023-11-02T08:43:14.184 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.184 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.184 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 2, 2023-11-02T08:43:14.184 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "chassis" 2023-11-02T08:43:14.185 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.185 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.185 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 3, 2023-11-02T08:43:14.185 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "rack" 2023-11-02T08:43:14.185 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.186 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.186 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 4, 2023-11-02T08:43:14.186 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "row" 2023-11-02T08:43:14.186 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.186 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.186 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 5, 2023-11-02T08:43:14.187 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "pdu" 2023-11-02T08:43:14.187 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.187 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.187 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 6, 2023-11-02T08:43:14.187 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "pod" 2023-11-02T08:43:14.188 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.188 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.188 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 7, 2023-11-02T08:43:14.188 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "room" 2023-11-02T08:43:14.188 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.188 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.189 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 8, 2023-11-02T08:43:14.189 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "datacenter" 2023-11-02T08:43:14.189 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.189 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.189 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 9, 2023-11-02T08:43:14.189 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "zone" 2023-11-02T08:43:14.190 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.190 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.190 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 10, 2023-11-02T08:43:14.190 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "region" 2023-11-02T08:43:14.190 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.191 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.191 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:43:14.191 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "root" 2023-11-02T08:43:14.191 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.191 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:14.191 INFO:tasks.workunit.client.0.smithi116.stdout: "buckets": [ 2023-11-02T08:43:14.192 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.192 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -1, 2023-11-02T08:43:14.192 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "default", 2023-11-02T08:43:14.192 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:43:14.192 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "root", 2023-11-02T08:43:14.193 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:43:14.193 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:43:14.193 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:43:14.193 INFO:tasks.workunit.client.0.smithi116.stderr:report 197129320 2023-11-02T08:43:14.194 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:43:14.194 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.194 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -3, 2023-11-02T08:43:14.195 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:43:14.195 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:43:14.195 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.195 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.195 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.195 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.196 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -2, 2023-11-02T08:43:14.196 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "default~hdd", 2023-11-02T08:43:14.196 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:43:14.196 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "root", 2023-11-02T08:43:14.196 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:43:14.197 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:43:14.197 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:43:14.197 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:43:14.197 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.197 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -4, 2023-11-02T08:43:14.197 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:43:14.198 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:43:14.198 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.198 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.198 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.198 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.199 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -3, 2023-11-02T08:43:14.199 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "smithi116", 2023-11-02T08:43:14.199 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:43:14.199 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "host", 2023-11-02T08:43:14.199 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:43:14.199 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:43:14.200 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:43:14.200 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:43:14.200 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.200 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:43:14.200 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:43:14.201 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:43:14.201 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.201 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.201 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:43:14.201 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:43:14.201 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 1 2023-11-02T08:43:14.202 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.202 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.202 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:43:14.202 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:43:14.202 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 2 2023-11-02T08:43:14.203 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.203 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.203 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.203 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.203 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -4, 2023-11-02T08:43:14.204 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "smithi116~hdd", 2023-11-02T08:43:14.204 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:43:14.204 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "host", 2023-11-02T08:43:14.204 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:43:14.204 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:43:14.204 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:43:14.205 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:43:14.205 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.205 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:43:14.205 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:43:14.205 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:43:14.205 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.206 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.206 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:43:14.206 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:43:14.206 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 1 2023-11-02T08:43:14.206 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.207 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.207 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:43:14.207 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:43:14.207 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 2 2023-11-02T08:43:14.207 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.207 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.208 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.208 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:14.208 INFO:tasks.workunit.client.0.smithi116.stdout: "rules": [ 2023-11-02T08:43:14.208 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.208 INFO:tasks.workunit.client.0.smithi116.stdout: "rule_id": 0, 2023-11-02T08:43:14.209 INFO:tasks.workunit.client.0.smithi116.stdout: "rule_name": "replicated_rule", 2023-11-02T08:43:14.209 INFO:tasks.workunit.client.0.smithi116.stdout: "type": 1, 2023-11-02T08:43:14.209 INFO:tasks.workunit.client.0.smithi116.stdout: "steps": [ 2023-11-02T08:43:14.209 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.209 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "take", 2023-11-02T08:43:14.209 INFO:tasks.workunit.client.0.smithi116.stdout: "item": -1, 2023-11-02T08:43:14.210 INFO:tasks.workunit.client.0.smithi116.stdout: "item_name": "default" 2023-11-02T08:43:14.210 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.210 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.210 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "choose_firstn", 2023-11-02T08:43:14.210 INFO:tasks.workunit.client.0.smithi116.stdout: "num": 0, 2023-11-02T08:43:14.211 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "osd" 2023-11-02T08:43:14.211 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.211 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.211 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "emit" 2023-11-02T08:43:14.211 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.211 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:14.212 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.212 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:14.212 INFO:tasks.workunit.client.0.smithi116.stdout: "tunables": { 2023-11-02T08:43:14.212 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_local_tries": 0, 2023-11-02T08:43:14.212 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_local_fallback_tries": 0, 2023-11-02T08:43:14.213 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_total_tries": 50, 2023-11-02T08:43:14.213 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_descend_once": 1, 2023-11-02T08:43:14.213 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_vary_r": 1, 2023-11-02T08:43:14.213 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_stable": 1, 2023-11-02T08:43:14.213 INFO:tasks.workunit.client.0.smithi116.stdout: "straw_calc_version": 1, 2023-11-02T08:43:14.213 INFO:tasks.workunit.client.0.smithi116.stdout: "allowed_bucket_algs": 54, 2023-11-02T08:43:14.214 INFO:tasks.workunit.client.0.smithi116.stdout: "profile": "jewel", 2023-11-02T08:43:14.214 INFO:tasks.workunit.client.0.smithi116.stdout: "optimal_tunables": 1, 2023-11-02T08:43:14.214 INFO:tasks.workunit.client.0.smithi116.stdout: "legacy_tunables": 0, 2023-11-02T08:43:14.214 INFO:tasks.workunit.client.0.smithi116.stdout: "minimum_required_version": "jewel", 2023-11-02T08:43:14.214 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables": 1, 2023-11-02T08:43:14.215 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables2": 1, 2023-11-02T08:43:14.215 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v2_rules": 0, 2023-11-02T08:43:14.215 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables3": 1, 2023-11-02T08:43:14.215 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v3_rules": 0, 2023-11-02T08:43:14.215 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v4_buckets": 1, 2023-11-02T08:43:14.215 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables5": 1, 2023-11-02T08:43:14.216 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v5_rules": 0 2023-11-02T08:43:14.216 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.216 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_args": {} 2023-11-02T08:43:14.216 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.216 INFO:tasks.workunit.client.0.smithi116.stdout: "fsmap": { 2023-11-02T08:43:14.217 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 1, 2023-11-02T08:43:14.217 INFO:tasks.workunit.client.0.smithi116.stdout: "default_fscid": -1, 2023-11-02T08:43:14.217 INFO:tasks.workunit.client.0.smithi116.stdout: "compat": { 2023-11-02T08:43:14.217 INFO:tasks.workunit.client.0.smithi116.stdout: "compat": {}, 2023-11-02T08:43:14.217 INFO:tasks.workunit.client.0.smithi116.stdout: "ro_compat": {}, 2023-11-02T08:43:14.217 INFO:tasks.workunit.client.0.smithi116.stdout: "incompat": { 2023-11-02T08:43:14.218 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_1": "base v0.20", 2023-11-02T08:43:14.218 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_2": "client writeable ranges", 2023-11-02T08:43:14.218 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_3": "default file layouts on dirs", 2023-11-02T08:43:14.218 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_4": "dir inode in separate object", 2023-11-02T08:43:14.218 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_5": "mds uses versioned encoding", 2023-11-02T08:43:14.219 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_6": "dirfrag is stored in omap", 2023-11-02T08:43:14.219 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_8": "no anchor table", 2023-11-02T08:43:14.219 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_9": "file layout v2", 2023-11-02T08:43:14.219 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_10": "snaprealm v2" 2023-11-02T08:43:14.219 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.219 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.220 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_flags": { 2023-11-02T08:43:14.220 INFO:tasks.workunit.client.0.smithi116.stdout: "enable_multiple": true, 2023-11-02T08:43:14.220 INFO:tasks.workunit.client.0.smithi116.stdout: "ever_enabled_multiple": true 2023-11-02T08:43:14.220 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.220 INFO:tasks.workunit.client.0.smithi116.stdout: "standbys": [], 2023-11-02T08:43:14.220 INFO:tasks.workunit.client.0.smithi116.stdout: "filesystems": [] 2023-11-02T08:43:14.221 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.221 INFO:tasks.workunit.client.0.smithi116.stdout: "mdsmap_first_committed": 1, 2023-11-02T08:43:14.221 INFO:tasks.workunit.client.0.smithi116.stdout: "mdsmap_last_committed": 1, 2023-11-02T08:43:14.221 INFO:tasks.workunit.client.0.smithi116.stdout: "auth": { 2023-11-02T08:43:14.221 INFO:tasks.workunit.client.0.smithi116.stdout: "first_committed": 1, 2023-11-02T08:43:14.222 INFO:tasks.workunit.client.0.smithi116.stdout: "last_committed": 5, 2023-11-02T08:43:14.222 INFO:tasks.workunit.client.0.smithi116.stdout: "num_secrets": 3 2023-11-02T08:43:14.222 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.222 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg": 32, 2023-11-02T08:43:14.222 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_active": 32, 2023-11-02T08:43:14.222 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_unknown": 0, 2023-11-02T08:43:14.223 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osd": 3, 2023-11-02T08:43:14.223 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_sum": { 2023-11-02T08:43:14.223 INFO:tasks.workunit.client.0.smithi116.stdout: "stat_sum": { 2023-11-02T08:43:14.223 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes": 46804, 2023-11-02T08:43:14.223 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects": 96, 2023-11-02T08:43:14.224 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_clones": 0, 2023-11-02T08:43:14.224 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_copies": 288, 2023-11-02T08:43:14.224 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing_on_primary": 0, 2023-11-02T08:43:14.224 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing": 0, 2023-11-02T08:43:14.224 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_degraded": 96, 2023-11-02T08:43:14.224 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_misplaced": 0, 2023-11-02T08:43:14.225 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_unfound": 0, 2023-11-02T08:43:14.225 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_dirty": 96, 2023-11-02T08:43:14.225 INFO:tasks.workunit.client.0.smithi116.stdout: "num_whiteouts": 0, 2023-11-02T08:43:14.225 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read": 0, 2023-11-02T08:43:14.225 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read_kb": 0, 2023-11-02T08:43:14.226 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write": 90, 2023-11-02T08:43:14.226 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write_kb": 90, 2023-11-02T08:43:14.226 INFO:tasks.workunit.client.0.smithi116.stdout: "num_scrub_errors": 0, 2023-11-02T08:43:14.226 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shallow_scrub_errors": 0, 2023-11-02T08:43:14.226 INFO:tasks.workunit.client.0.smithi116.stdout: "num_deep_scrub_errors": 0, 2023-11-02T08:43:14.226 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_recovered": 0, 2023-11-02T08:43:14.227 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_recovered": 0, 2023-11-02T08:43:14.227 INFO:tasks.workunit.client.0.smithi116.stdout: "num_keys_recovered": 0, 2023-11-02T08:43:14.227 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_omap": 0, 2023-11-02T08:43:14.227 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_hit_set_archive": 0, 2023-11-02T08:43:14.227 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_hit_set_archive": 0, 2023-11-02T08:43:14.228 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush": 0, 2023-11-02T08:43:14.228 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_kb": 0, 2023-11-02T08:43:14.228 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict": 0, 2023-11-02T08:43:14.228 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_kb": 0, 2023-11-02T08:43:14.228 INFO:tasks.workunit.client.0.smithi116.stdout: "num_promote": 0, 2023-11-02T08:43:14.228 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_high": 0, 2023-11-02T08:43:14.229 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_low": 0, 2023-11-02T08:43:14.229 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_some": 0, 2023-11-02T08:43:14.229 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_full": 0, 2023-11-02T08:43:14.229 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_pinned": 0, 2023-11-02T08:43:14.229 INFO:tasks.workunit.client.0.smithi116.stdout: "num_legacy_snapsets": 0, 2023-11-02T08:43:14.229 INFO:tasks.workunit.client.0.smithi116.stdout: "num_large_omap_objects": 0, 2023-11-02T08:43:14.230 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_manifest": 0, 2023-11-02T08:43:14.230 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_bytes": 0, 2023-11-02T08:43:14.230 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_keys": 0, 2023-11-02T08:43:14.230 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_repaired": 0 2023-11-02T08:43:14.230 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.231 INFO:tasks.workunit.client.0.smithi116.stdout: "store_stats": { 2023-11-02T08:43:14.231 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 0, 2023-11-02T08:43:14.231 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 0, 2023-11-02T08:43:14.231 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:43:14.231 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 0, 2023-11-02T08:43:14.232 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 0, 2023-11-02T08:43:14.232 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:43:14.232 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:43:14.232 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:43:14.232 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 0, 2023-11-02T08:43:14.232 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 0 2023-11-02T08:43:14.233 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.233 INFO:tasks.workunit.client.0.smithi116.stdout: "log_size": 419, 2023-11-02T08:43:14.233 INFO:tasks.workunit.client.0.smithi116.stdout: "ondisk_log_size": 419, 2023-11-02T08:43:14.233 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 64, 2023-11-02T08:43:14.233 INFO:tasks.workunit.client.0.smithi116.stdout: "acting": 64, 2023-11-02T08:43:14.233 INFO:tasks.workunit.client.0.smithi116.stdout: "num_store_stats": 0 2023-11-02T08:43:14.234 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.234 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_sum": { 2023-11-02T08:43:14.234 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 0, 2023-11-02T08:43:14.234 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 0, 2023-11-02T08:43:14.234 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pgs": 96, 2023-11-02T08:43:14.235 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osds": 3, 2023-11-02T08:43:14.235 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_osds": 3, 2023-11-02T08:43:14.235 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_omap_osds": 3, 2023-11-02T08:43:14.235 INFO:tasks.workunit.client.0.smithi116.stdout: "kb": 314572800, 2023-11-02T08:43:14.235 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used": 84280, 2023-11-02T08:43:14.235 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_data": 2592, 2023-11-02T08:43:14.236 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_omap": 2, 2023-11-02T08:43:14.236 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_meta": 81661, 2023-11-02T08:43:14.236 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_avail": 314488520, 2023-11-02T08:43:14.236 INFO:tasks.workunit.client.0.smithi116.stdout: "statfs": { 2023-11-02T08:43:14.236 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 322122547200, 2023-11-02T08:43:14.237 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 322036244480, 2023-11-02T08:43:14.237 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:43:14.237 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 2654208, 2023-11-02T08:43:14.237 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 608720, 2023-11-02T08:43:14.237 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:43:14.237 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:43:14.238 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:43:14.238 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 2366, 2023-11-02T08:43:14.238 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 83621570 2023-11-02T08:43:14.238 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.238 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_peers": [], 2023-11-02T08:43:14.239 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_trim_queue_len": 0, 2023-11-02T08:43:14.239 INFO:tasks.workunit.client.0.smithi116.stdout: "num_snap_trimming": 0, 2023-11-02T08:43:14.239 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shards_repaired": 0, 2023-11-02T08:43:14.239 INFO:tasks.workunit.client.0.smithi116.stdout: "op_queue_age_hist": { 2023-11-02T08:43:14.239 INFO:tasks.workunit.client.0.smithi116.stdout: "histogram": [], 2023-11-02T08:43:14.239 INFO:tasks.workunit.client.0.smithi116.stdout: "upper_bound": 1 2023-11-02T08:43:14.240 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.240 INFO:tasks.workunit.client.0.smithi116.stdout: "perf_stat": { 2023-11-02T08:43:14.240 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ms": 143, 2023-11-02T08:43:14.240 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ms": 143, 2023-11-02T08:43:14.240 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ns": 143000000, 2023-11-02T08:43:14.240 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ns": 143000000 2023-11-02T08:43:14.241 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.241 INFO:tasks.workunit.client.0.smithi116.stdout: "alerts": [], 2023-11-02T08:43:14.241 INFO:tasks.workunit.client.0.smithi116.stdout: "network_ping_times": [] 2023-11-02T08:43:14.241 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.241 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_sum_by_class": { 2023-11-02T08:43:14.242 INFO:tasks.workunit.client.0.smithi116.stdout: "hdd": { 2023-11-02T08:43:14.242 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 0, 2023-11-02T08:43:14.242 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 0, 2023-11-02T08:43:14.242 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pgs": 96, 2023-11-02T08:43:14.242 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osds": 3, 2023-11-02T08:43:14.242 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_osds": 3, 2023-11-02T08:43:14.243 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_omap_osds": 3, 2023-11-02T08:43:14.243 INFO:tasks.workunit.client.0.smithi116.stdout: "kb": 314572800, 2023-11-02T08:43:14.243 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used": 84280, 2023-11-02T08:43:14.243 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_data": 2592, 2023-11-02T08:43:14.243 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_omap": 2, 2023-11-02T08:43:14.244 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_meta": 81661, 2023-11-02T08:43:14.244 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_avail": 314488520, 2023-11-02T08:43:14.244 INFO:tasks.workunit.client.0.smithi116.stdout: "statfs": { 2023-11-02T08:43:14.244 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 322122547200, 2023-11-02T08:43:14.244 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 322036244480, 2023-11-02T08:43:14.245 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:43:14.245 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 2654208, 2023-11-02T08:43:14.245 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 608720, 2023-11-02T08:43:14.245 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:43:14.245 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:43:14.245 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:43:14.246 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 2366, 2023-11-02T08:43:14.246 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 83621570 2023-11-02T08:43:14.246 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.246 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_peers": [], 2023-11-02T08:43:14.246 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_trim_queue_len": 0, 2023-11-02T08:43:14.247 INFO:tasks.workunit.client.0.smithi116.stdout: "num_snap_trimming": 0, 2023-11-02T08:43:14.247 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shards_repaired": 0, 2023-11-02T08:43:14.247 INFO:tasks.workunit.client.0.smithi116.stdout: "op_queue_age_hist": { 2023-11-02T08:43:14.247 INFO:tasks.workunit.client.0.smithi116.stdout: "histogram": [], 2023-11-02T08:43:14.247 INFO:tasks.workunit.client.0.smithi116.stdout: "upper_bound": 1 2023-11-02T08:43:14.247 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.248 INFO:tasks.workunit.client.0.smithi116.stdout: "perf_stat": { 2023-11-02T08:43:14.248 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ms": 143, 2023-11-02T08:43:14.248 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ms": 143, 2023-11-02T08:43:14.248 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ns": 143000000, 2023-11-02T08:43:14.248 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ns": 143000000 2023-11-02T08:43:14.248 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.249 INFO:tasks.workunit.client.0.smithi116.stdout: "alerts": [], 2023-11-02T08:43:14.249 INFO:tasks.workunit.client.0.smithi116.stdout: "network_ping_times": [] 2023-11-02T08:43:14.249 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.249 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.249 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_stats": [ 2023-11-02T08:43:14.250 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.250 INFO:tasks.workunit.client.0.smithi116.stdout: "poolid": 1, 2023-11-02T08:43:14.250 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg": 32, 2023-11-02T08:43:14.250 INFO:tasks.workunit.client.0.smithi116.stdout: "stat_sum": { 2023-11-02T08:43:14.250 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes": 46804, 2023-11-02T08:43:14.250 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects": 96, 2023-11-02T08:43:14.251 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_clones": 0, 2023-11-02T08:43:14.251 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_copies": 288, 2023-11-02T08:43:14.251 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing_on_primary": 0, 2023-11-02T08:43:14.251 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing": 0, 2023-11-02T08:43:14.251 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_degraded": 96, 2023-11-02T08:43:14.252 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_misplaced": 0, 2023-11-02T08:43:14.252 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_unfound": 0, 2023-11-02T08:43:14.252 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_dirty": 96, 2023-11-02T08:43:14.252 INFO:tasks.workunit.client.0.smithi116.stdout: "num_whiteouts": 0, 2023-11-02T08:43:14.252 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read": 0, 2023-11-02T08:43:14.252 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read_kb": 0, 2023-11-02T08:43:14.253 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write": 90, 2023-11-02T08:43:14.253 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write_kb": 90, 2023-11-02T08:43:14.253 INFO:tasks.workunit.client.0.smithi116.stdout: "num_scrub_errors": 0, 2023-11-02T08:43:14.253 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shallow_scrub_errors": 0, 2023-11-02T08:43:14.253 INFO:tasks.workunit.client.0.smithi116.stdout: "num_deep_scrub_errors": 0, 2023-11-02T08:43:14.254 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_recovered": 0, 2023-11-02T08:43:14.254 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_recovered": 0, 2023-11-02T08:43:14.254 INFO:tasks.workunit.client.0.smithi116.stdout: "num_keys_recovered": 0, 2023-11-02T08:43:14.254 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_omap": 0, 2023-11-02T08:43:14.254 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_hit_set_archive": 0, 2023-11-02T08:43:14.254 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_hit_set_archive": 0, 2023-11-02T08:43:14.255 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush": 0, 2023-11-02T08:43:14.255 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_kb": 0, 2023-11-02T08:43:14.255 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict": 0, 2023-11-02T08:43:14.255 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_kb": 0, 2023-11-02T08:43:14.255 INFO:tasks.workunit.client.0.smithi116.stdout: "num_promote": 0, 2023-11-02T08:43:14.256 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_high": 0, 2023-11-02T08:43:14.256 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_low": 0, 2023-11-02T08:43:14.256 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_some": 0, 2023-11-02T08:43:14.256 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_full": 0, 2023-11-02T08:43:14.256 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_pinned": 0, 2023-11-02T08:43:14.256 INFO:tasks.workunit.client.0.smithi116.stdout: "num_legacy_snapsets": 0, 2023-11-02T08:43:14.257 INFO:tasks.workunit.client.0.smithi116.stdout: "num_large_omap_objects": 0, 2023-11-02T08:43:14.257 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_manifest": 0, 2023-11-02T08:43:14.257 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_bytes": 0, 2023-11-02T08:43:14.257 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_keys": 0, 2023-11-02T08:43:14.257 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_repaired": 0 2023-11-02T08:43:14.257 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.258 INFO:tasks.workunit.client.0.smithi116.stdout: "store_stats": { 2023-11-02T08:43:14.258 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 0, 2023-11-02T08:43:14.258 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 0, 2023-11-02T08:43:14.258 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:43:14.258 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 1220608, 2023-11-02T08:43:14.259 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 143228, 2023-11-02T08:43:14.259 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:43:14.259 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:43:14.259 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:43:14.259 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 0, 2023-11-02T08:43:14.259 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 0 2023-11-02T08:43:14.260 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.260 INFO:tasks.workunit.client.0.smithi116.stdout: "log_size": 419, 2023-11-02T08:43:14.260 INFO:tasks.workunit.client.0.smithi116.stdout: "ondisk_log_size": 419, 2023-11-02T08:43:14.260 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 64, 2023-11-02T08:43:14.260 INFO:tasks.workunit.client.0.smithi116.stdout: "acting": 64, 2023-11-02T08:43:14.261 INFO:tasks.workunit.client.0.smithi116.stdout: "num_store_stats": 3 2023-11-02T08:43:14.261 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.261 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:14.261 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_stats": [ 2023-11-02T08:43:14.261 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.261 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:43:14.262 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 231928233995 2023-11-02T08:43:14.262 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.262 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.263 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:43:14.263 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 42949672988 2023-11-02T08:43:14.263 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.263 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.263 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:43:14.264 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 249108103178 2023-11-02T08:43:14.264 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.264 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:14.264 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_by_state": [ 2023-11-02T08:43:14.264 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.265 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "active+undersized+degraded", 2023-11-02T08:43:14.265 INFO:tasks.workunit.client.0.smithi116.stdout: "num": 32 2023-11-02T08:43:14.265 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.265 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:14.266 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_by_osd": [ 2023-11-02T08:43:14.266 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.266 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:43:14.266 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 18, 2023-11-02T08:43:14.266 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 32, 2023-11-02T08:43:14.267 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:43:14.267 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.267 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.267 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:43:14.267 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 14, 2023-11-02T08:43:14.268 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 32, 2023-11-02T08:43:14.268 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:43:14.268 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.268 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.268 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:43:14.269 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 0, 2023-11-02T08:43:14.269 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 0, 2023-11-02T08:43:14.269 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:43:14.269 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.269 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:14.270 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snaps": [ 2023-11-02T08:43:14.270 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:14.270 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": 1, 2023-11-02T08:43:14.270 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snaps": {} 2023-11-02T08:43:14.271 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.271 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:14.271 INFO:tasks.workunit.client.0.smithi116.stdout: "servicemap": { 2023-11-02T08:43:14.271 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 4, 2023-11-02T08:43:14.271 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:42:04.821182+0000", 2023-11-02T08:43:14.272 INFO:tasks.workunit.client.0.smithi116.stdout: "services": { 2023-11-02T08:43:14.272 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": { 2023-11-02T08:43:14.272 INFO:tasks.workunit.client.0.smithi116.stdout: "daemons": { 2023-11-02T08:43:14.272 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": "", 2023-11-02T08:43:14.272 INFO:tasks.workunit.client.0.smithi116.stdout: "2": { 2023-11-02T08:43:14.273 INFO:tasks.workunit.client.0.smithi116.stdout: "start_epoch": 0, 2023-11-02T08:43:14.273 INFO:tasks.workunit.client.0.smithi116.stdout: "start_stamp": "0.000000", 2023-11-02T08:43:14.273 INFO:tasks.workunit.client.0.smithi116.stdout: "gid": 0, 2023-11-02T08:43:14.273 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "(unrecognized address family 0)/0", 2023-11-02T08:43:14.273 INFO:tasks.workunit.client.0.smithi116.stdout: "metadata": {}, 2023-11-02T08:43:14.274 INFO:tasks.workunit.client.0.smithi116.stdout: "task_status": {} 2023-11-02T08:43:14.274 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.274 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.274 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.274 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.275 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:14.275 INFO:tasks.workunit.client.0.smithi116.stdout: "mgrstat_first_committed": 1, 2023-11-02T08:43:14.275 INFO:tasks.workunit.client.0.smithi116.stdout: "mgrstat_last_committed": 87, 2023-11-02T08:43:14.275 INFO:tasks.workunit.client.0.smithi116.stdout: "logm_first_committed": 1, 2023-11-02T08:43:14.275 INFO:tasks.workunit.client.0.smithi116.stdout: "logm_last_committed": 397, 2023-11-02T08:43:14.276 INFO:tasks.workunit.client.0.smithi116.stdout: "paxos": { 2023-11-02T08:43:14.276 INFO:tasks.workunit.client.0.smithi116.stdout: "first_committed": 1, 2023-11-02T08:43:14.276 INFO:tasks.workunit.client.0.smithi116.stdout: "last_committed": 576, 2023-11-02T08:43:14.276 INFO:tasks.workunit.client.0.smithi116.stdout: "last_pn": 0, 2023-11-02T08:43:14.276 INFO:tasks.workunit.client.0.smithi116.stdout: "accepted_pn": 0 2023-11-02T08:43:14.277 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:14.277 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T08:43:14.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2023-11-02T08:43:14.278 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: expr 2 + 2 2023-11-02T08:43:14.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: objname=existing_4 2023-11-02T08:43:14.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2023-11-02T08:43:14.279 INFO:tasks.workunit.client.0.smithi116.stdout:writing non-divergent object existing_4 2023-11-02T08:43:14.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2023-11-02T08:43:14.424 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:43:14.425 INFO:tasks.workunit.client.0.smithi116.stdout:1.1f 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945032+0000 32'97 66:129 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.425 INFO:tasks.workunit.client.0.smithi116.stdout:1.1e 4 0 4 0 0 1947 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.946255+0000 34'98 66:153 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.425 INFO:tasks.workunit.client.0.smithi116.stdout:1.1d 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944739+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.425 INFO:tasks.workunit.client.0.smithi116.stdout:1.b 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920318+0000 32'97 66:132 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.426 INFO:tasks.workunit.client.0.smithi116.stdout:1.a 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944603+0000 32'97 66:120 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.426 INFO:tasks.workunit.client.0.smithi116.stdout:1.9 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920339+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.426 INFO:tasks.workunit.client.0.smithi116.stdout:1.8 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945695+0000 32'97 66:120 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.426 INFO:tasks.workunit.client.0.smithi116.stdout:1.7 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.851862+0000 32'97 66:129 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.427 INFO:tasks.workunit.client.0.smithi116.stdout:1.6 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.946582+0000 32'97 66:141 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.427 INFO:tasks.workunit.client.0.smithi116.stdout:1.5 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945458+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.427 INFO:tasks.workunit.client.0.smithi116.stdout:1.4 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944062+0000 32'97 66:132 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.427 INFO:tasks.workunit.client.0.smithi116.stdout:1.3 4 0 4 0 0 1948 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.947103+0000 34'98 66:141 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.427 INFO:tasks.workunit.client.0.smithi116.stdout:1.2 1 0 1 0 0 483 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944339+0000 32'97 66:153 [0,2] 0 [0,2] 0 32'97 2023-11-02T08:42:12.415700+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T18:58:39.256358+0000 1 0 2023-11-02T08:43:14.428 INFO:tasks.workunit.client.0.smithi116.stdout:1.c 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920311+0000 32'97 66:132 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.428 INFO:tasks.workunit.client.0.smithi116.stdout:1.1 2 0 2 0 0 966 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944180+0000 32'97 66:143 [2,0] 2 [2,0] 2 32'97 2023-11-02T08:42:09.981464+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T14:45:22.323682+0000 2 0 2023-11-02T08:43:14.428 INFO:tasks.workunit.client.0.smithi116.stdout:1.1c 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944438+0000 32'97 66:132 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.428 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.947058+0000 32'97 66:132 [0,2] 0 [0,2] 0 32'97 2023-11-02T08:42:07.198292+0000 32'97 2023-11-02T08:42:07.198292+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.429 INFO:tasks.workunit.client.0.smithi116.stdout:1.d 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.946518+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.429 INFO:tasks.workunit.client.0.smithi116.stdout:1.e 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945332+0000 32'97 66:141 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.429 INFO:tasks.workunit.client.0.smithi116.stdout:1.f 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.946682+0000 32'97 66:129 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.429 INFO:tasks.workunit.client.0.smithi116.stdout:1.10 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944393+0000 32'97 66:132 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.430 INFO:tasks.workunit.client.0.smithi116.stdout:1.11 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944108+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.430 INFO:tasks.workunit.client.0.smithi116.stdout:1.12 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944113+0000 32'97 66:141 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.430 INFO:tasks.workunit.client.0.smithi116.stdout:1.13 3 0 3 0 0 1449 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920751+0000 32'97 66:141 [2,0] 2 [2,0] 2 32'97 2023-11-02T08:42:10.923269+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T08:52:50.221029+0000 3 0 2023-11-02T08:43:14.430 INFO:tasks.workunit.client.0.smithi116.stdout:1.14 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.947335+0000 32'97 66:132 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.431 INFO:tasks.workunit.client.0.smithi116.stdout:1.15 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.947192+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.431 INFO:tasks.workunit.client.0.smithi116.stdout:1.16 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944598+0000 32'97 66:141 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.431 INFO:tasks.workunit.client.0.smithi116.stdout:1.17 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945003+0000 32'97 66:129 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.431 INFO:tasks.workunit.client.0.smithi116.stdout:1.18 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944425+0000 32'97 66:120 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.431 INFO:tasks.workunit.client.0.smithi116.stdout:1.19 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944351+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.432 INFO:tasks.workunit.client.0.smithi116.stdout:1.1a 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920724+0000 32'97 66:120 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.432 INFO:tasks.workunit.client.0.smithi116.stdout:1.1b 4 0 4 0 0 1947 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.920740+0000 34'98 66:143 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:14.432 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:43:14.432 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:43:14.433 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:43:14.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2023-11-02T08:43:14.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2023-11-02T08:43:14.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: wait_for_clean 2023-11-02T08:43:14.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T08:43:14.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T08:43:14.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T08:43:14.525 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2023-11-02T08:43:14.526 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T08:43:14.526 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T08:43:14.527 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T08:43:14.527 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T08:43:14.527 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T08:43:14.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2023-11-02T08:43:14.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T08:43:14.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T08:43:14.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T08:43:14.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:43:14.577 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:43:14.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:43:14.993 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:43:14.993 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:43:14.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:43:14.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:43:14.994 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:43:15.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=231928233997 2023-11-02T08:43:15.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 231928233997 2023-11-02T08:43:15.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-231928233997' 2023-11-02T08:43:15.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:43:15.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:43:15.237 INFO:tasks.workunit.client.0.smithi116.stderr:Error ENXIO: problem getting command descriptions from osd.1 2023-11-02T08:43:15.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2023-11-02T08:43:15.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2023-11-02T08:43:15.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2023-11-02T08:43:15.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:43:15.239 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:43:15.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=249108103181 2023-11-02T08:43:15.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 249108103181 2023-11-02T08:43:15.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-231928233997 2-249108103181' 2023-11-02T08:43:15.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:43:15.359 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-231928233997 2023-11-02T08:43:15.359 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:43:15.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:43:15.360 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-231928233997 2023-11-02T08:43:15.360 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:43:15.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=231928233997 2023-11-02T08:43:15.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 231928233997' 2023-11-02T08:43:15.361 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 231928233997 2023-11-02T08:43:15.362 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:43:15.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 231928233995 -lt 231928233997 2023-11-02T08:43:15.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T08:43:16.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T08:43:16.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:43:17.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 231928233998 -lt 231928233997 2023-11-02T08:43:17.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:43:17.185 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-249108103181 2023-11-02T08:43:17.185 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:43:17.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:43:17.186 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-249108103181 2023-11-02T08:43:17.186 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:43:17.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=249108103181 2023-11-02T08:43:17.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 249108103181' 2023-11-02T08:43:17.187 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 249108103181 2023-11-02T08:43:17.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:43:17.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 249108103181 -lt 249108103181 2023-11-02T08:43:17.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T08:43:17.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:43:17.607 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:43:18.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2023-11-02T08:43:18.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:43:18.121 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:43:18.121 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:43:18.122 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:43:18.122 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:43:18.122 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:43:18.122 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:43:18.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:43:18.518 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:43:18.519 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:43:18.519 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:43:19.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2023-11-02T08:43:19.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2023-11-02T08:43:19.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2023-11-02T08:43:19.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2023-11-02T08:43:19.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:43:19.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T08:43:19.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:43:19.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:43:19.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:43:19.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:43:19.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:43:19.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:43:19.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:43:19.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:43:19.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:43:19.524 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:43:19.524 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:43:19.525 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:43:20.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2023-11-02T08:43:20.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:43:20.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:43:20.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:43:20.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:43:20.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:43:20.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:43:20.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:43:20.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:43:20.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:43:20.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:43:20.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2023-11-02T08:43:20.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:43:20.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2023-11-02T08:43:20.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:43:20.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:43:20.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:43:20.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:43:20.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:43:20.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:43:20.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:43:20.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:43:21.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:43:21.143 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:43:21.143 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:43:21.143 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:43:21.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2023-11-02T08:43:21.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:43:21.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:43:21.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:43:21.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:43:21.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:43:21.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:43:21.652 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:43:21.653 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:43:22.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:43:22.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:43:22.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2023-11-02T08:43:22.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:43:22.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2023-11-02T08:43:22.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:43:22.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:43:22.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:43:22.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:43:22.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:43:22.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:43:22.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:43:22.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:43:22.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:43:22.968 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:43:22.968 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:43:22.968 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:43:23.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2023-11-02T08:43:23.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:43:23.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:43:23.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:43:23.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:43:23.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:43:23.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:43:23.504 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:43:23.504 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:43:24.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:43:24.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:43:24.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2023-11-02T08:43:24.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:43:24.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2023-11-02T08:43:24.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:43:24.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:43:24.845 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:43:24.845 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:43:24.845 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:43:24.845 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:43:24.846 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:43:24.846 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:43:25.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:43:25.224 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:43:25.224 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:43:25.225 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:43:25.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2023-11-02T08:43:25.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:43:25.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:43:25.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:43:25.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:43:25.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:43:25.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:43:25.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:43:25.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:43:26.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:43:26.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:43:26.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2023-11-02T08:43:26.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:43:26.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2023-11-02T08:43:27.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:43:27.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:43:27.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:43:27.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:43:27.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:43:27.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:43:27.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:43:27.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:43:28.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:43:28.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:43:28.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:43:28.248 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:43:28.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2023-11-02T08:43:28.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:43:28.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:43:28.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:43:28.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:43:28.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:43:28.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:43:28.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:43:28.751 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:43:29.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:43:29.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:43:29.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2023-11-02T08:43:29.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:43:29.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2023-11-02T08:43:32.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:43:32.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:43:32.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:43:32.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:43:32.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:43:32.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:43:32.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:43:32.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:43:32.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:43:32.838 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:43:32.838 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:43:32.838 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:43:33.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2023-11-02T08:43:33.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:43:33.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:43:33.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:43:33.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:43:33.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:43:33.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:43:33.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:43:33.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:43:33.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:43:33.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:43:33.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2023-11-02T08:43:33.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:43:33.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2023-11-02T08:43:40.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:43:40.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:43:40.246 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:43:40.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:43:40.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:43:40.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:43:40.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:43:40.248 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:43:40.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:43:40.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:43:40.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:43:40.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:43:41.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2023-11-02T08:43:41.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:43:41.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:43:41.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:43:41.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:43:41.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:43:41.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:43:41.167 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:43:41.167 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:43:41.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:43:41.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:43:41.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2023-11-02T08:43:41.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:43:41.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2023-11-02T08:43:48.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:43:48.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:43:48.972 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:43:48.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:43:48.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:43:48.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:43:48.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:43:48.974 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:43:49.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:43:49.367 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:43:49.367 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:43:49.368 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:43:49.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2023-11-02T08:43:49.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:43:49.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:43:49.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:43:49.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:43:49.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:43:49.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:43:49.878 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:43:49.878 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:43:50.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:43:50.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:43:50.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2023-11-02T08:43:50.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2023-11-02T08:43:50.932 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T08:43:50.932 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_fingerprint": "9265913a-476f-4246-8d00-7318d249d9e9", 2023-11-02T08:43:50.932 INFO:tasks.workunit.client.0.smithi116.stdout: "version": "18.2.0-1159-gb5a9486b", 2023-11-02T08:43:50.932 INFO:tasks.workunit.client.0.smithi116.stdout: "commit": "b5a9486b4a6f144f83c51b7ec33111a2c86dc081", 2023-11-02T08:43:50.933 INFO:tasks.workunit.client.0.smithi116.stdout: "timestamp": "2023-11-02T08:43:50.928325+0000", 2023-11-02T08:43:50.933 INFO:tasks.workunit.client.0.smithi116.stdout: "tag": "", 2023-11-02T08:43:50.933 INFO:tasks.workunit.client.0.smithi116.stdout: "health": { 2023-11-02T08:43:50.933 INFO:tasks.workunit.client.0.smithi116.stdout: "status": "HEALTH_WARN", 2023-11-02T08:43:50.933 INFO:tasks.workunit.client.0.smithi116.stdout: "checks": { 2023-11-02T08:43:50.933 INFO:tasks.workunit.client.0.smithi116.stdout: "OSDMAP_FLAGS": { 2023-11-02T08:43:50.934 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:43:50.934 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:43:50.934 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "nodown,noout,noin flag(s) set", 2023-11-02T08:43:50.934 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 17 2023-11-02T08:43:50.934 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.935 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [], 2023-11-02T08:43:50.935 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:43:50.935 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.935 INFO:tasks.workunit.client.0.smithi116.stdout: "OSD_DOWN": { 2023-11-02T08:43:50.935 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:43:50.936 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:43:50.936 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "1 osds down", 2023-11-02T08:43:50.936 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 1 2023-11-02T08:43:50.936 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.936 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:43:50.937 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.937 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "osd.1 (root=default,host=smithi116) is down" 2023-11-02T08:43:50.937 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:50.938 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:50.938 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:43:50.938 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.938 INFO:tasks.workunit.client.0.smithi116.stdout: "PG_DEGRADED": { 2023-11-02T08:43:50.938 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:43:50.939 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:43:50.939 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "Degraded data redundancy: 96/288 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2023-11-02T08:43:50.939 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 64 2023-11-02T08:43:50.939 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.940 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:43:50.940 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.940 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.0 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2023-11-02T08:43:50.940 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.940 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.940 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.1 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2023-11-02T08:43:50.941 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.941 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.941 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.2 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2023-11-02T08:43:50.941 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.941 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.942 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.3 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2023-11-02T08:43:50.942 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.942 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.942 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.4 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2023-11-02T08:43:50.942 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.943 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.943 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.5 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2023-11-02T08:43:50.943 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.943 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.943 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.6 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2023-11-02T08:43:50.944 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.944 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.944 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.7 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2023-11-02T08:43:50.944 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.944 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.945 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.8 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2023-11-02T08:43:50.945 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.945 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.945 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.9 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2023-11-02T08:43:50.945 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.945 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.946 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.a is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2023-11-02T08:43:50.946 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.946 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.946 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.b is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2023-11-02T08:43:50.946 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.947 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.947 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.c is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2023-11-02T08:43:50.947 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.947 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.948 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.d is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2023-11-02T08:43:50.948 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.948 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.948 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.e is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2023-11-02T08:43:50.949 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.949 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.949 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.f is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2023-11-02T08:43:50.949 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.949 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.950 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.10 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2023-11-02T08:43:50.950 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.950 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.950 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.11 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2023-11-02T08:43:50.950 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.951 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.951 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.12 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2023-11-02T08:43:50.951 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.951 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.951 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.13 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2023-11-02T08:43:50.951 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.952 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.952 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.14 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2023-11-02T08:43:50.952 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.952 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.952 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.15 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2023-11-02T08:43:50.953 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.953 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.953 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.16 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2023-11-02T08:43:50.953 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.953 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.953 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.17 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2023-11-02T08:43:50.954 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.954 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.954 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.18 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2023-11-02T08:43:50.954 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.954 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.955 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.19 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2023-11-02T08:43:50.955 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.955 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.955 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.1a is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2023-11-02T08:43:50.955 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.956 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.956 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.1b is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2023-11-02T08:43:50.956 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.956 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.956 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.1c is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2023-11-02T08:43:50.957 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.957 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.957 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.1d is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2023-11-02T08:43:50.957 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.957 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.957 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.1e is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2023-11-02T08:43:50.958 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.958 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.958 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.1f is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2023-11-02T08:43:50.959 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:50.959 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:50.959 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:43:50.959 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.959 INFO:tasks.workunit.client.0.smithi116.stdout: "POOL_APP_NOT_ENABLED": { 2023-11-02T08:43:50.960 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:43:50.960 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:43:50.960 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "1 pool(s) do not have an application enabled", 2023-11-02T08:43:50.960 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 1 2023-11-02T08:43:50.960 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.960 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:43:50.961 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.961 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "application not enabled on pool 'test'" 2023-11-02T08:43:50.961 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.961 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.961 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2023-11-02T08:43:50.962 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:50.962 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:50.962 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:43:50.962 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:50.962 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.963 INFO:tasks.workunit.client.0.smithi116.stdout: "mutes": [] 2023-11-02T08:43:50.963 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.963 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap_first_committed": 1, 2023-11-02T08:43:50.963 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap_last_committed": 1, 2023-11-02T08:43:50.963 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap": { 2023-11-02T08:43:50.963 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 1, 2023-11-02T08:43:50.964 INFO:tasks.workunit.client.0.smithi116.stdout: "fsid": "0c9a372f-cec7-47d9-a309-8248453aabeb", 2023-11-02T08:43:50.964 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:39:57.418791Z", 2023-11-02T08:43:50.964 INFO:tasks.workunit.client.0.smithi116.stdout: "created": "2023-11-02T08:39:57.418791Z", 2023-11-02T08:43:50.964 INFO:tasks.workunit.client.0.smithi116.stdout: "min_mon_release": 18, 2023-11-02T08:43:50.964 INFO:tasks.workunit.client.0.smithi116.stdout: "min_mon_release_name": "reef", 2023-11-02T08:43:50.965 INFO:tasks.workunit.client.0.smithi116.stdout: "election_strategy": 1, 2023-11-02T08:43:50.965 INFO:tasks.workunit.client.0.smithi116.stdout: "disallowed_leaders: ": "", 2023-11-02T08:43:50.965 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode": false, 2023-11-02T08:43:50.965 INFO:tasks.workunit.client.0.smithi116.stdout: "tiebreaker_mon": "", 2023-11-02T08:43:50.965 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_ranks: ": "", 2023-11-02T08:43:50.966 INFO:tasks.workunit.client.0.smithi116.stdout: "features": { 2023-11-02T08:43:50.966 INFO:tasks.workunit.client.0.smithi116.stdout: "persistent": [ 2023-11-02T08:43:50.966 INFO:tasks.workunit.client.0.smithi116.stdout: "kraken", 2023-11-02T08:43:50.966 INFO:tasks.workunit.client.0.smithi116.stdout: "luminous", 2023-11-02T08:43:50.966 INFO:tasks.workunit.client.0.smithi116.stdout: "mimic", 2023-11-02T08:43:50.966 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap-prune", 2023-11-02T08:43:50.967 INFO:tasks.workunit.client.0.smithi116.stdout: "nautilus", 2023-11-02T08:43:50.967 INFO:tasks.workunit.client.0.smithi116.stdout: "octopus", 2023-11-02T08:43:50.967 INFO:tasks.workunit.client.0.smithi116.stdout: "pacific", 2023-11-02T08:43:50.967 INFO:tasks.workunit.client.0.smithi116.stdout: "elector-pinging", 2023-11-02T08:43:50.967 INFO:tasks.workunit.client.0.smithi116.stdout: "quincy", 2023-11-02T08:43:50.968 INFO:tasks.workunit.client.0.smithi116.stdout: "reef" 2023-11-02T08:43:50.968 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:50.968 INFO:tasks.workunit.client.0.smithi116.stdout: "optional": [] 2023-11-02T08:43:50.968 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.969 INFO:tasks.workunit.client.0.smithi116.stdout: "mons": [ 2023-11-02T08:43:50.969 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.969 INFO:tasks.workunit.client.0.smithi116.stdout: "rank": 0, 2023-11-02T08:43:50.969 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "a", 2023-11-02T08:43:50.969 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:43:50.970 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:50.970 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.970 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:50.970 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:7115", 2023-11-02T08:43:50.970 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 0 2023-11-02T08:43:50.971 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:50.971 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:50.971 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.971 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:7115/0", 2023-11-02T08:43:50.971 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:7115/0", 2023-11-02T08:43:50.972 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 0, 2023-11-02T08:43:50.972 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 0, 2023-11-02T08:43:50.972 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_location": "{}" 2023-11-02T08:43:50.972 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:50.972 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:50.972 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.973 INFO:tasks.workunit.client.0.smithi116.stdout: "quorum": [ 2023-11-02T08:43:50.973 INFO:tasks.workunit.client.0.smithi116.stdout: 0 2023-11-02T08:43:50.973 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:50.973 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap": { 2023-11-02T08:43:50.973 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 66, 2023-11-02T08:43:50.974 INFO:tasks.workunit.client.0.smithi116.stdout: "fsid": "0c9a372f-cec7-47d9-a309-8248453aabeb", 2023-11-02T08:43:50.974 INFO:tasks.workunit.client.0.smithi116.stdout: "created": "2023-11-02T08:39:58.207786+0000", 2023-11-02T08:43:50.974 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:42:37.824722+0000", 2023-11-02T08:43:50.974 INFO:tasks.workunit.client.0.smithi116.stdout: "last_up_change": "2023-11-02T08:42:34.544550+0000", 2023-11-02T08:43:50.974 INFO:tasks.workunit.client.0.smithi116.stdout: "last_in_change": "2023-11-02T08:42:36.775521+0000", 2023-11-02T08:43:50.974 INFO:tasks.workunit.client.0.smithi116.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-11-02T08:43:50.975 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_num": 5800384, 2023-11-02T08:43:50.975 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_set": [ 2023-11-02T08:43:50.975 INFO:tasks.workunit.client.0.smithi116.stdout: "nodown", 2023-11-02T08:43:50.975 INFO:tasks.workunit.client.0.smithi116.stdout: "noin", 2023-11-02T08:43:50.975 INFO:tasks.workunit.client.0.smithi116.stdout: "noout", 2023-11-02T08:43:50.976 INFO:tasks.workunit.client.0.smithi116.stdout: "pglog_hardlimit", 2023-11-02T08:43:50.976 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snapdirs", 2023-11-02T08:43:50.976 INFO:tasks.workunit.client.0.smithi116.stdout: "recovery_deletes", 2023-11-02T08:43:50.976 INFO:tasks.workunit.client.0.smithi116.stdout: "sortbitwise" 2023-11-02T08:43:50.976 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:50.976 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_version": 9, 2023-11-02T08:43:50.977 INFO:tasks.workunit.client.0.smithi116.stdout: "full_ratio": 0.99000000953674316, 2023-11-02T08:43:50.977 INFO:tasks.workunit.client.0.smithi116.stdout: "backfillfull_ratio": 0.99000000953674316, 2023-11-02T08:43:50.977 INFO:tasks.workunit.client.0.smithi116.stdout: "nearfull_ratio": 0.85000002384185791, 2023-11-02T08:43:50.977 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_snapshot": "", 2023-11-02T08:43:50.977 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_max": 1, 2023-11-02T08:43:50.978 INFO:tasks.workunit.client.0.smithi116.stdout: "max_osd": 3, 2023-11-02T08:43:50.978 INFO:tasks.workunit.client.0.smithi116.stdout: "require_min_compat_client": "luminous", 2023-11-02T08:43:50.978 INFO:tasks.workunit.client.0.smithi116.stdout: "min_compat_client": "jewel", 2023-11-02T08:43:50.978 INFO:tasks.workunit.client.0.smithi116.stdout: "require_osd_release": "reef", 2023-11-02T08:43:50.978 INFO:tasks.workunit.client.0.smithi116.stdout: "allow_crimson": false, 2023-11-02T08:43:50.978 INFO:tasks.workunit.client.0.smithi116.stdout: "pools": [ 2023-11-02T08:43:50.979 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.979 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": 1, 2023-11-02T08:43:50.979 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_name": "test", 2023-11-02T08:43:50.979 INFO:tasks.workunit.client.0.smithi116.stdout: "create_time": "2023-11-02T08:41:38.449621+0000", 2023-11-02T08:43:50.979 INFO:tasks.workunit.client.0.smithi116.stdout: "flags": 1, 2023-11-02T08:43:50.980 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_names": "hashpspool", 2023-11-02T08:43:50.980 INFO:tasks.workunit.client.0.smithi116.stdout: "type": 1, 2023-11-02T08:43:50.980 INFO:tasks.workunit.client.0.smithi116.stdout: "size": 3, 2023-11-02T08:43:50.980 INFO:tasks.workunit.client.0.smithi116.stdout: "min_size": 2, 2023-11-02T08:43:50.980 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_rule": 0, 2023-11-02T08:43:50.980 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_count": 0, 2023-11-02T08:43:50.981 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_target": 0, 2023-11-02T08:43:50.981 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_barrier": 0, 2023-11-02T08:43:50.981 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-11-02T08:43:50.981 INFO:tasks.workunit.client.0.smithi116.stdout: "object_hash": 2, 2023-11-02T08:43:50.981 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_autoscale_mode": "on", 2023-11-02T08:43:50.982 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num": 32, 2023-11-02T08:43:50.982 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_placement_num": 14, 2023-11-02T08:43:50.982 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_placement_num_target": 32, 2023-11-02T08:43:50.982 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num_target": 32, 2023-11-02T08:43:50.982 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num_pending": 32, 2023-11-02T08:43:50.983 INFO:tasks.workunit.client.0.smithi116.stdout: "last_pg_merge_meta": { 2023-11-02T08:43:50.983 INFO:tasks.workunit.client.0.smithi116.stdout: "source_pgid": "0.0", 2023-11-02T08:43:50.983 INFO:tasks.workunit.client.0.smithi116.stdout: "ready_epoch": 0, 2023-11-02T08:43:50.983 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_started": 0, 2023-11-02T08:43:50.983 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_clean": 0, 2023-11-02T08:43:50.983 INFO:tasks.workunit.client.0.smithi116.stdout: "source_version": "0'0", 2023-11-02T08:43:50.984 INFO:tasks.workunit.client.0.smithi116.stdout: "target_version": "0'0" 2023-11-02T08:43:50.984 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.984 INFO:tasks.workunit.client.0.smithi116.stdout: "last_change": "64", 2023-11-02T08:43:50.984 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend": "0", 2023-11-02T08:43:50.984 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend_prenautilus": "0", 2023-11-02T08:43:50.985 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend_preluminous": "33", 2023-11-02T08:43:50.985 INFO:tasks.workunit.client.0.smithi116.stdout: "auid": 0, 2023-11-02T08:43:50.985 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_mode": "selfmanaged", 2023-11-02T08:43:50.985 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_seq": 0, 2023-11-02T08:43:50.985 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_epoch": 0, 2023-11-02T08:43:50.985 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_snaps": [], 2023-11-02T08:43:50.986 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_snaps": "[]", 2023-11-02T08:43:50.986 INFO:tasks.workunit.client.0.smithi116.stdout: "quota_max_bytes": 0, 2023-11-02T08:43:50.986 INFO:tasks.workunit.client.0.smithi116.stdout: "quota_max_objects": 0, 2023-11-02T08:43:50.986 INFO:tasks.workunit.client.0.smithi116.stdout: "tiers": [], 2023-11-02T08:43:50.986 INFO:tasks.workunit.client.0.smithi116.stdout: "tier_of": -1, 2023-11-02T08:43:50.987 INFO:tasks.workunit.client.0.smithi116.stdout: "read_tier": -1, 2023-11-02T08:43:50.987 INFO:tasks.workunit.client.0.smithi116.stdout: "write_tier": -1, 2023-11-02T08:43:50.987 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_mode": "none", 2023-11-02T08:43:50.987 INFO:tasks.workunit.client.0.smithi116.stdout: "target_max_bytes": 0, 2023-11-02T08:43:50.987 INFO:tasks.workunit.client.0.smithi116.stdout: "target_max_objects": 0, 2023-11-02T08:43:50.987 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-11-02T08:43:50.988 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-11-02T08:43:50.988 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_full_ratio_micro": 800000, 2023-11-02T08:43:50.988 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_min_flush_age": 0, 2023-11-02T08:43:50.988 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_min_evict_age": 0, 2023-11-02T08:43:50.988 INFO:tasks.workunit.client.0.smithi116.stdout: "erasure_code_profile": "", 2023-11-02T08:43:50.988 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_params": { 2023-11-02T08:43:50.989 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "none" 2023-11-02T08:43:50.989 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.989 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_period": 0, 2023-11-02T08:43:50.989 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_count": 0, 2023-11-02T08:43:50.989 INFO:tasks.workunit.client.0.smithi116.stdout: "use_gmt_hitset": true, 2023-11-02T08:43:50.990 INFO:tasks.workunit.client.0.smithi116.stdout: "min_read_recency_for_promote": 0, 2023-11-02T08:43:50.990 INFO:tasks.workunit.client.0.smithi116.stdout: "min_write_recency_for_promote": 0, 2023-11-02T08:43:50.990 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_grade_decay_rate": 0, 2023-11-02T08:43:50.990 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_search_last_n": 0, 2023-11-02T08:43:50.990 INFO:tasks.workunit.client.0.smithi116.stdout: "grade_table": [], 2023-11-02T08:43:50.990 INFO:tasks.workunit.client.0.smithi116.stdout: "stripe_width": 0, 2023-11-02T08:43:50.991 INFO:tasks.workunit.client.0.smithi116.stdout: "expected_num_objects": 0, 2023-11-02T08:43:50.991 INFO:tasks.workunit.client.0.smithi116.stdout: "fast_read": false, 2023-11-02T08:43:50.991 INFO:tasks.workunit.client.0.smithi116.stdout: "options": {}, 2023-11-02T08:43:50.991 INFO:tasks.workunit.client.0.smithi116.stdout: "application_metadata": {}, 2023-11-02T08:43:50.991 INFO:tasks.workunit.client.0.smithi116.stdout: "read_balance": { 2023-11-02T08:43:50.992 INFO:tasks.workunit.client.0.smithi116.stdout: "score_acting": 1.690000057220459, 2023-11-02T08:43:50.992 INFO:tasks.workunit.client.0.smithi116.stdout: "score_stable": 1.690000057220459, 2023-11-02T08:43:50.992 INFO:tasks.workunit.client.0.smithi116.stdout: "optimal_score": 0.67000001668930054, 2023-11-02T08:43:50.992 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_score_acting": 1.1299999952316284, 2023-11-02T08:43:50.992 INFO:tasks.workunit.client.0.smithi116.stdout: "raw_score_stable": 1.1299999952316284, 2023-11-02T08:43:50.992 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity_weighted": 1, 2023-11-02T08:43:50.993 INFO:tasks.workunit.client.0.smithi116.stdout: "average_primary_affinity": 1, 2023-11-02T08:43:50.993 INFO:tasks.workunit.client.0.smithi116.stdout: "average_primary_affinity_weighted": 1 2023-11-02T08:43:50.993 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:50.993 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:50.993 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:50.994 INFO:tasks.workunit.client.0.smithi116.stdout: "osds": [ 2023-11-02T08:43:50.994 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.994 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:43:50.994 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "584ee92b-5cfd-43f0-8535-9a70b0612348", 2023-11-02T08:43:50.994 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 1, 2023-11-02T08:43:50.994 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:43:50.995 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:43:50.995 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:43:50.995 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 5, 2023-11-02T08:43:50.995 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 38, 2023-11-02T08:43:50.995 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 54, 2023-11-02T08:43:50.996 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 63, 2023-11-02T08:43:50.996 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 45, 2023-11-02T08:43:50.996 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:43:50.996 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:43:50.996 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:50.996 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.997 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:50.997 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6802", 2023-11-02T08:43:50.997 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 120543 2023-11-02T08:43:50.997 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.997 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:50.998 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:50.998 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6803", 2023-11-02T08:43:50.998 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 120543 2023-11-02T08:43:50.998 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:50.998 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:50.998 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:50.999 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:43:50.999 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:50.999 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.000 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:51.000 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6804", 2023-11-02T08:43:51.000 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 120543 2023-11-02T08:43:51.000 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.000 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.000 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:51.001 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6805", 2023-11-02T08:43:51.001 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 120543 2023-11-02T08:43:51.001 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.001 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.002 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.002 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:43:51.002 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:51.003 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.003 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:51.003 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6808", 2023-11-02T08:43:51.003 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 120543 2023-11-02T08:43:51.004 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.004 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.004 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:51.004 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6809", 2023-11-02T08:43:51.004 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 120543 2023-11-02T08:43:51.005 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.005 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.005 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.005 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:43:51.005 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:51.005 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.006 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:51.006 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6806", 2023-11-02T08:43:51.006 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 120543 2023-11-02T08:43:51.006 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.006 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.007 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:51.007 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6807", 2023-11-02T08:43:51.007 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 120543 2023-11-02T08:43:51.007 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.007 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.007 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.008 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6803/120543", 2023-11-02T08:43:51.008 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6805/120543", 2023-11-02T08:43:51.008 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6809/120543", 2023-11-02T08:43:51.008 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6807/120543", 2023-11-02T08:43:51.008 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:43:51.009 INFO:tasks.workunit.client.0.smithi116.stdout: "exists", 2023-11-02T08:43:51.009 INFO:tasks.workunit.client.0.smithi116.stdout: "up" 2023-11-02T08:43:51.009 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.009 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.009 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.010 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:43:51.010 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "4d681ad5-42d6-443f-ac67-86739e3ca916", 2023-11-02T08:43:51.010 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 0, 2023-11-02T08:43:51.010 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:43:51.010 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:43:51.010 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:43:51.011 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 0, 2023-11-02T08:43:51.011 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 0, 2023-11-02T08:43:51.011 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 10, 2023-11-02T08:43:51.011 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 43, 2023-11-02T08:43:51.011 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 46, 2023-11-02T08:43:51.012 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:43:51.012 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:43:51.012 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:51.012 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.012 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:51.013 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6810", 2023-11-02T08:43:51.013 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 114539 2023-11-02T08:43:51.013 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.013 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.013 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:51.013 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6811", 2023-11-02T08:43:51.014 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 114539 2023-11-02T08:43:51.014 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.014 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.014 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.014 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:43:51.015 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:51.015 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.015 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:51.015 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6812", 2023-11-02T08:43:51.015 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 114539 2023-11-02T08:43:51.015 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.016 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.016 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:51.016 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6813", 2023-11-02T08:43:51.016 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 114539 2023-11-02T08:43:51.017 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.018 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.018 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.018 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:43:51.018 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:51.019 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.019 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:51.019 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6816", 2023-11-02T08:43:51.019 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 114539 2023-11-02T08:43:51.019 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.020 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.020 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:51.020 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6817", 2023-11-02T08:43:51.020 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 114539 2023-11-02T08:43:51.020 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.021 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.021 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.021 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:43:51.021 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:51.021 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.021 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:51.022 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6814", 2023-11-02T08:43:51.022 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 114539 2023-11-02T08:43:51.022 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.022 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.023 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:51.023 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6815", 2023-11-02T08:43:51.023 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 114539 2023-11-02T08:43:51.024 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.024 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.024 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.024 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6811/114539", 2023-11-02T08:43:51.024 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6813/114539", 2023-11-02T08:43:51.025 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6817/114539", 2023-11-02T08:43:51.025 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6815/114539", 2023-11-02T08:43:51.025 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:43:51.025 INFO:tasks.workunit.client.0.smithi116.stdout: "exists" 2023-11-02T08:43:51.025 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.026 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.026 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.027 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:43:51.027 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "8b4576fe-bdf3-4241-80eb-7d5e8439887c", 2023-11-02T08:43:51.027 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 1, 2023-11-02T08:43:51.027 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:43:51.027 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:43:51.028 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:43:51.028 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 15, 2023-11-02T08:43:51.028 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 39, 2023-11-02T08:43:51.028 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 58, 2023-11-02T08:43:51.028 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 65, 2023-11-02T08:43:51.029 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 48, 2023-11-02T08:43:51.029 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:43:51.029 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:43:51.029 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:51.029 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.029 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:51.030 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6810", 2023-11-02T08:43:51.030 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 121087 2023-11-02T08:43:51.030 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.030 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.030 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:51.031 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6811", 2023-11-02T08:43:51.031 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 121087 2023-11-02T08:43:51.031 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.031 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.031 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.032 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:43:51.032 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:51.032 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.032 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:51.032 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6812", 2023-11-02T08:43:51.032 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 121087 2023-11-02T08:43:51.033 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.033 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.033 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:51.033 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6813", 2023-11-02T08:43:51.033 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 121087 2023-11-02T08:43:51.034 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.034 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.034 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.034 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:43:51.034 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:51.034 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.035 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:51.035 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6816", 2023-11-02T08:43:51.035 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 121087 2023-11-02T08:43:51.035 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.035 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.036 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:51.036 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6817", 2023-11-02T08:43:51.036 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 121087 2023-11-02T08:43:51.036 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.036 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.036 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.037 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:43:51.037 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:43:51.037 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.037 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:43:51.037 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6814", 2023-11-02T08:43:51.038 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 121087 2023-11-02T08:43:51.038 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.038 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.038 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:43:51.038 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6815", 2023-11-02T08:43:51.038 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 121087 2023-11-02T08:43:51.039 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.039 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.039 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.039 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6811/121087", 2023-11-02T08:43:51.040 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6813/121087", 2023-11-02T08:43:51.040 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6817/121087", 2023-11-02T08:43:51.040 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6815/121087", 2023-11-02T08:43:51.040 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:43:51.040 INFO:tasks.workunit.client.0.smithi116.stdout: "exists", 2023-11-02T08:43:51.041 INFO:tasks.workunit.client.0.smithi116.stdout: "up" 2023-11-02T08:43:51.041 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.041 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.041 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:51.041 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_xinfo": [ 2023-11-02T08:43:51.041 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.042 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:43:51.042 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:42:23.925605+0000", 2023-11-02T08:43:51.042 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:43:51.042 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:43:51.042 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:43:51.043 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:43:51.043 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:40:28.898043+0000", 2023-11-02T08:43:51.043 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:43:51.043 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.043 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.043 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:43:51.044 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:42:24.478710+0000", 2023-11-02T08:43:51.044 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:43:51.044 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:43:51.044 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:43:51.044 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:43:51.045 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:41:02.215113+0000", 2023-11-02T08:43:51.045 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:43:51.045 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.045 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.045 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:43:51.045 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:42:25.101538+0000", 2023-11-02T08:43:51.046 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:43:51.046 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:43:51.046 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:43:51.046 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:43:51.046 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:41:35.175518+0000", 2023-11-02T08:43:51.047 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:43:51.047 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.047 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:51.047 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap": [], 2023-11-02T08:43:51.047 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap_items": [], 2023-11-02T08:43:51.047 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap_primaries": [], 2023-11-02T08:43:51.048 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_temp": [], 2023-11-02T08:43:51.048 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_temp": [], 2023-11-02T08:43:51.048 INFO:tasks.workunit.client.0.smithi116.stdout: "blocklist": {}, 2023-11-02T08:43:51.048 INFO:tasks.workunit.client.0.smithi116.stdout: "range_blocklist": {}, 2023-11-02T08:43:51.049 INFO:tasks.workunit.client.0.smithi116.stdout: "erasure_code_profiles": { 2023-11-02T08:43:51.049 INFO:tasks.workunit.client.0.smithi116.stdout: "default": { 2023-11-02T08:43:51.049 INFO:tasks.workunit.client.0.smithi116.stdout: "k": "2", 2023-11-02T08:43:51.049 INFO:tasks.workunit.client.0.smithi116.stdout: "m": "2", 2023-11-02T08:43:51.049 INFO:tasks.workunit.client.0.smithi116.stdout: "plugin": "jerasure", 2023-11-02T08:43:51.050 INFO:tasks.workunit.client.0.smithi116.stdout: "technique": "reed_sol_van" 2023-11-02T08:43:51.050 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.050 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.050 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_snaps_queue": [], 2023-11-02T08:43:51.050 INFO:tasks.workunit.client.0.smithi116.stdout: "new_removed_snaps": [], 2023-11-02T08:43:51.050 INFO:tasks.workunit.client.0.smithi116.stdout: "new_purged_snaps": [], 2023-11-02T08:43:51.051 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_node_flags": {}, 2023-11-02T08:43:51.051 INFO:tasks.workunit.client.0.smithi116.stdout: "device_class_flags": {}, 2023-11-02T08:43:51.051 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode": { 2023-11-02T08:43:51.051 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode_enabled": false, 2023-11-02T08:43:51.051 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_bucket_count": 0, 2023-11-02T08:43:51.052 INFO:tasks.workunit.client.0.smithi116.stdout: "degraded_stretch_mode": 0, 2023-11-02T08:43:51.052 INFO:tasks.workunit.client.0.smithi116.stdout: "recovering_stretch_mode": 0, 2023-11-02T08:43:51.052 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode_bucket": 0 2023-11-02T08:43:51.052 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.052 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.052 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_metadata": [ 2023-11-02T08:43:51.053 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.053 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:43:51.053 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:43:51.053 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6804/120543,v1:127.0.0.1:6805/120543]", 2023-11-02T08:43:51.053 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:43:51.054 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:43:51.054 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:43:51.054 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:43:51.054 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:43:51.054 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:43:51.055 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:43:51.055 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:43:51.055 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:43:51.055 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:43:51.056 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2023-11-02T08:43:51.056 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:43:51.056 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:43:51.056 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:43:51.056 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:43:51.057 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:43:51.057 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:43:51.057 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:43:51.057 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:43:51.057 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:43:51.057 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:43:51.058 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:40:04.089233Z", 2023-11-02T08:43:51.058 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:43:51.058 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:43:51.058 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:43:51.058 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:43:51.059 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:43:51.059 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:43:51.059 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:43:51.059 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6802/120543,v1:127.0.0.1:6803/120543]", 2023-11-02T08:43:51.059 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:43:51.059 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/120543,v1:127.0.0.1:6809/120543]", 2023-11-02T08:43:51.060 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/120543,v1:127.0.0.1:6807/120543]", 2023-11-02T08:43:51.060 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:43:51.060 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:43:51.060 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:43:51.060 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:43:51.061 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:43:51.061 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:43:51.061 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:43:51.061 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:43:51.061 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:43:51.061 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/0", 2023-11-02T08:43:51.062 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:43:51.062 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:43:51.062 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:43:51.062 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.062 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.063 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:43:51.063 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:43:51.063 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6812/114539,v1:127.0.0.1:6813/114539]", 2023-11-02T08:43:51.063 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:43:51.063 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:43:51.064 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:43:51.064 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:43:51.064 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:43:51.064 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:43:51.065 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:43:51.065 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:43:51.065 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:43:51.065 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:43:51.065 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2023-11-02T08:43:51.066 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:43:51.066 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:43:51.066 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:43:51.066 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:43:51.066 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:43:51.066 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:43:51.067 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:43:51.067 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:43:51.067 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:43:51.067 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:43:51.067 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:40:36.033676Z", 2023-11-02T08:43:51.068 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:43:51.068 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:43:51.068 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:43:51.068 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:43:51.068 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:43:51.068 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:43:51.069 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:43:51.069 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6810/114539,v1:127.0.0.1:6811/114539]", 2023-11-02T08:43:51.069 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:43:51.069 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/114539,v1:127.0.0.1:6817/114539]", 2023-11-02T08:43:51.069 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/114539,v1:127.0.0.1:6815/114539]", 2023-11-02T08:43:51.070 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:43:51.070 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:43:51.070 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:43:51.070 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:43:51.071 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:43:51.071 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:43:51.071 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:43:51.071 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:43:51.071 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:43:51.072 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/1", 2023-11-02T08:43:51.072 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:43:51.072 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:43:51.072 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:43:51.073 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.073 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.073 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:43:51.073 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:43:51.073 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6812/121087,v1:127.0.0.1:6813/121087]", 2023-11-02T08:43:51.074 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:43:51.074 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:43:51.074 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:43:51.074 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:43:51.074 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:43:51.074 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:43:51.075 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:43:51.075 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:43:51.075 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:43:51.075 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:43:51.075 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2023-11-02T08:43:51.076 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:43:51.076 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:43:51.076 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:43:51.076 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:43:51.076 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:43:51.076 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:43:51.077 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:43:51.077 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:43:51.077 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:43:51.077 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:43:51.077 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:41:08.750575Z", 2023-11-02T08:43:51.078 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:43:51.078 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:43:51.078 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:43:51.078 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:43:51.078 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:43:51.078 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:43:51.079 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:43:51.079 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6810/121087,v1:127.0.0.1:6811/121087]", 2023-11-02T08:43:51.079 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:43:51.079 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/121087,v1:127.0.0.1:6817/121087]", 2023-11-02T08:43:51.079 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/121087,v1:127.0.0.1:6815/121087]", 2023-11-02T08:43:51.080 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:43:51.080 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:43:51.080 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:43:51.080 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:43:51.080 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:43:51.080 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:43:51.081 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:43:51.081 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:43:51.081 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:43:51.081 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/2", 2023-11-02T08:43:51.081 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:43:51.082 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:43:51.082 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:43:51.082 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.082 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:51.082 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_clean_epochs": { 2023-11-02T08:43:51.082 INFO:tasks.workunit.client.0.smithi116.stdout: "min_last_epoch_clean": 0, 2023-11-02T08:43:51.083 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_clean": { 2023-11-02T08:43:51.083 INFO:tasks.workunit.client.0.smithi116.stdout: "per_pool": [ 2023-11-02T08:43:51.083 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.083 INFO:tasks.workunit.client.0.smithi116.stdout: "poolid": 1, 2023-11-02T08:43:51.083 INFO:tasks.workunit.client.0.smithi116.stdout: "floor": 36 2023-11-02T08:43:51.084 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.084 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.084 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.084 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_epochs": [ 2023-11-02T08:43:51.084 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.084 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:43:51.085 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 55 2023-11-02T08:43:51.085 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.085 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.085 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:43:51.085 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 10 2023-11-02T08:43:51.086 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.086 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.086 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:43:51.086 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 59 2023-11-02T08:43:51.086 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.086 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.087 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.087 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_first_committed": 1, 2023-11-02T08:43:51.087 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_last_committed": 66, 2023-11-02T08:43:51.087 INFO:tasks.workunit.client.0.smithi116.stdout: "crushmap": { 2023-11-02T08:43:51.087 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": [ 2023-11-02T08:43:51.088 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.088 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:43:51.088 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.0", 2023-11-02T08:43:51.088 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:43:51.088 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.088 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.089 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:43:51.089 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.1", 2023-11-02T08:43:51.089 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:43:51.089 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.089 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.089 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:43:51.090 INFO:tasks.workunit.client.0.smithi116.stderr:report 602494827 2023-11-02T08:43:51.091 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.2", 2023-11-02T08:43:51.091 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:43:51.091 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.091 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:51.091 INFO:tasks.workunit.client.0.smithi116.stdout: "types": [ 2023-11-02T08:43:51.092 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.092 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 0, 2023-11-02T08:43:51.092 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd" 2023-11-02T08:43:51.092 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.092 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.092 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:43:51.093 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "host" 2023-11-02T08:43:51.093 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.093 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.093 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 2, 2023-11-02T08:43:51.093 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "chassis" 2023-11-02T08:43:51.094 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.094 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.094 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 3, 2023-11-02T08:43:51.094 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "rack" 2023-11-02T08:43:51.094 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.094 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.095 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 4, 2023-11-02T08:43:51.095 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "row" 2023-11-02T08:43:51.095 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.095 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.095 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 5, 2023-11-02T08:43:51.096 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "pdu" 2023-11-02T08:43:51.096 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.096 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.096 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 6, 2023-11-02T08:43:51.096 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "pod" 2023-11-02T08:43:51.097 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.097 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.097 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 7, 2023-11-02T08:43:51.097 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "room" 2023-11-02T08:43:51.097 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.097 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.098 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 8, 2023-11-02T08:43:51.098 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "datacenter" 2023-11-02T08:43:51.098 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.098 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.098 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 9, 2023-11-02T08:43:51.099 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "zone" 2023-11-02T08:43:51.099 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.099 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.099 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 10, 2023-11-02T08:43:51.099 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "region" 2023-11-02T08:43:51.099 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.100 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.100 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:43:51.100 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "root" 2023-11-02T08:43:51.100 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.100 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:51.101 INFO:tasks.workunit.client.0.smithi116.stdout: "buckets": [ 2023-11-02T08:43:51.101 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.101 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -1, 2023-11-02T08:43:51.101 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "default", 2023-11-02T08:43:51.101 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:43:51.102 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "root", 2023-11-02T08:43:51.102 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:43:51.102 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:43:51.102 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:43:51.102 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:43:51.102 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.103 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -3, 2023-11-02T08:43:51.103 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:43:51.103 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:43:51.103 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.103 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.104 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.104 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.104 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -2, 2023-11-02T08:43:51.104 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "default~hdd", 2023-11-02T08:43:51.104 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:43:51.104 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "root", 2023-11-02T08:43:51.105 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:43:51.105 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:43:51.105 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:43:51.105 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:43:51.105 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.106 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -4, 2023-11-02T08:43:51.106 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:43:51.106 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:43:51.106 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.106 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.106 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.107 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.107 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -3, 2023-11-02T08:43:51.107 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "smithi116", 2023-11-02T08:43:51.107 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:43:51.107 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "host", 2023-11-02T08:43:51.108 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:43:51.108 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:43:51.108 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:43:51.108 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:43:51.108 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.108 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:43:51.109 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:43:51.109 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:43:51.109 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.109 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.109 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:43:51.109 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:43:51.110 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 1 2023-11-02T08:43:51.110 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.110 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.110 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:43:51.110 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:43:51.111 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 2 2023-11-02T08:43:51.111 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.111 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.111 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.111 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.113 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -4, 2023-11-02T08:43:51.113 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "smithi116~hdd", 2023-11-02T08:43:51.113 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:43:51.113 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "host", 2023-11-02T08:43:51.114 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:43:51.114 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:43:51.114 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:43:51.114 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:43:51.114 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.115 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:43:51.115 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:43:51.115 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:43:51.115 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.115 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.115 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:43:51.116 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:43:51.116 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 1 2023-11-02T08:43:51.116 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.116 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.116 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:43:51.117 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:43:51.117 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 2 2023-11-02T08:43:51.117 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.117 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.118 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.118 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:51.118 INFO:tasks.workunit.client.0.smithi116.stdout: "rules": [ 2023-11-02T08:43:51.118 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.119 INFO:tasks.workunit.client.0.smithi116.stdout: "rule_id": 0, 2023-11-02T08:43:51.119 INFO:tasks.workunit.client.0.smithi116.stdout: "rule_name": "replicated_rule", 2023-11-02T08:43:51.119 INFO:tasks.workunit.client.0.smithi116.stdout: "type": 1, 2023-11-02T08:43:51.119 INFO:tasks.workunit.client.0.smithi116.stdout: "steps": [ 2023-11-02T08:43:51.119 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.120 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "take", 2023-11-02T08:43:51.120 INFO:tasks.workunit.client.0.smithi116.stdout: "item": -1, 2023-11-02T08:43:51.120 INFO:tasks.workunit.client.0.smithi116.stdout: "item_name": "default" 2023-11-02T08:43:51.120 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.120 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.120 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "choose_firstn", 2023-11-02T08:43:51.121 INFO:tasks.workunit.client.0.smithi116.stdout: "num": 0, 2023-11-02T08:43:51.121 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "osd" 2023-11-02T08:43:51.121 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.121 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.121 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "emit" 2023-11-02T08:43:51.122 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.122 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:43:51.122 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.122 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:51.122 INFO:tasks.workunit.client.0.smithi116.stdout: "tunables": { 2023-11-02T08:43:51.123 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_local_tries": 0, 2023-11-02T08:43:51.123 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_local_fallback_tries": 0, 2023-11-02T08:43:51.123 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_total_tries": 50, 2023-11-02T08:43:51.123 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_descend_once": 1, 2023-11-02T08:43:51.123 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_vary_r": 1, 2023-11-02T08:43:51.124 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_stable": 1, 2023-11-02T08:43:51.124 INFO:tasks.workunit.client.0.smithi116.stdout: "straw_calc_version": 1, 2023-11-02T08:43:51.124 INFO:tasks.workunit.client.0.smithi116.stdout: "allowed_bucket_algs": 54, 2023-11-02T08:43:51.124 INFO:tasks.workunit.client.0.smithi116.stdout: "profile": "jewel", 2023-11-02T08:43:51.124 INFO:tasks.workunit.client.0.smithi116.stdout: "optimal_tunables": 1, 2023-11-02T08:43:51.125 INFO:tasks.workunit.client.0.smithi116.stdout: "legacy_tunables": 0, 2023-11-02T08:43:51.125 INFO:tasks.workunit.client.0.smithi116.stdout: "minimum_required_version": "jewel", 2023-11-02T08:43:51.125 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables": 1, 2023-11-02T08:43:51.125 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables2": 1, 2023-11-02T08:43:51.125 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v2_rules": 0, 2023-11-02T08:43:51.125 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables3": 1, 2023-11-02T08:43:51.126 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v3_rules": 0, 2023-11-02T08:43:51.126 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v4_buckets": 1, 2023-11-02T08:43:51.126 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables5": 1, 2023-11-02T08:43:51.126 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v5_rules": 0 2023-11-02T08:43:51.126 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.127 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_args": {} 2023-11-02T08:43:51.127 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.127 INFO:tasks.workunit.client.0.smithi116.stdout: "fsmap": { 2023-11-02T08:43:51.127 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 1, 2023-11-02T08:43:51.127 INFO:tasks.workunit.client.0.smithi116.stdout: "default_fscid": -1, 2023-11-02T08:43:51.128 INFO:tasks.workunit.client.0.smithi116.stdout: "compat": { 2023-11-02T08:43:51.128 INFO:tasks.workunit.client.0.smithi116.stdout: "compat": {}, 2023-11-02T08:43:51.128 INFO:tasks.workunit.client.0.smithi116.stdout: "ro_compat": {}, 2023-11-02T08:43:51.128 INFO:tasks.workunit.client.0.smithi116.stdout: "incompat": { 2023-11-02T08:43:51.128 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_1": "base v0.20", 2023-11-02T08:43:51.128 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_2": "client writeable ranges", 2023-11-02T08:43:51.129 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_3": "default file layouts on dirs", 2023-11-02T08:43:51.129 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_4": "dir inode in separate object", 2023-11-02T08:43:51.129 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_5": "mds uses versioned encoding", 2023-11-02T08:43:51.129 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_6": "dirfrag is stored in omap", 2023-11-02T08:43:51.129 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_8": "no anchor table", 2023-11-02T08:43:51.130 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_9": "file layout v2", 2023-11-02T08:43:51.130 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_10": "snaprealm v2" 2023-11-02T08:43:51.130 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.130 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.130 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_flags": { 2023-11-02T08:43:51.130 INFO:tasks.workunit.client.0.smithi116.stdout: "enable_multiple": true, 2023-11-02T08:43:51.131 INFO:tasks.workunit.client.0.smithi116.stdout: "ever_enabled_multiple": true 2023-11-02T08:43:51.131 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.131 INFO:tasks.workunit.client.0.smithi116.stdout: "standbys": [], 2023-11-02T08:43:51.131 INFO:tasks.workunit.client.0.smithi116.stdout: "filesystems": [] 2023-11-02T08:43:51.131 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.132 INFO:tasks.workunit.client.0.smithi116.stdout: "mdsmap_first_committed": 1, 2023-11-02T08:43:51.132 INFO:tasks.workunit.client.0.smithi116.stdout: "mdsmap_last_committed": 1, 2023-11-02T08:43:51.132 INFO:tasks.workunit.client.0.smithi116.stdout: "auth": { 2023-11-02T08:43:51.132 INFO:tasks.workunit.client.0.smithi116.stdout: "first_committed": 1, 2023-11-02T08:43:51.132 INFO:tasks.workunit.client.0.smithi116.stdout: "last_committed": 5, 2023-11-02T08:43:51.132 INFO:tasks.workunit.client.0.smithi116.stdout: "num_secrets": 3 2023-11-02T08:43:51.133 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.133 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg": 32, 2023-11-02T08:43:51.133 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_active": 32, 2023-11-02T08:43:51.133 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_unknown": 0, 2023-11-02T08:43:51.133 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osd": 3, 2023-11-02T08:43:51.134 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_sum": { 2023-11-02T08:43:51.134 INFO:tasks.workunit.client.0.smithi116.stdout: "stat_sum": { 2023-11-02T08:43:51.134 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes": 46451, 2023-11-02T08:43:51.134 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects": 96, 2023-11-02T08:43:51.134 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_clones": 0, 2023-11-02T08:43:51.134 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_copies": 288, 2023-11-02T08:43:51.135 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing_on_primary": 0, 2023-11-02T08:43:51.135 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing": 0, 2023-11-02T08:43:51.135 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_degraded": 96, 2023-11-02T08:43:51.135 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_misplaced": 0, 2023-11-02T08:43:51.135 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_unfound": 0, 2023-11-02T08:43:51.136 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_dirty": 96, 2023-11-02T08:43:51.136 INFO:tasks.workunit.client.0.smithi116.stdout: "num_whiteouts": 0, 2023-11-02T08:43:51.136 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read": 0, 2023-11-02T08:43:51.136 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read_kb": 0, 2023-11-02T08:43:51.136 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write": 91, 2023-11-02T08:43:51.136 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write_kb": 91, 2023-11-02T08:43:51.137 INFO:tasks.workunit.client.0.smithi116.stdout: "num_scrub_errors": 0, 2023-11-02T08:43:51.137 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shallow_scrub_errors": 0, 2023-11-02T08:43:51.137 INFO:tasks.workunit.client.0.smithi116.stdout: "num_deep_scrub_errors": 0, 2023-11-02T08:43:51.137 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_recovered": 0, 2023-11-02T08:43:51.137 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_recovered": 0, 2023-11-02T08:43:51.138 INFO:tasks.workunit.client.0.smithi116.stdout: "num_keys_recovered": 0, 2023-11-02T08:43:51.138 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_omap": 0, 2023-11-02T08:43:51.138 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_hit_set_archive": 0, 2023-11-02T08:43:51.138 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_hit_set_archive": 0, 2023-11-02T08:43:51.138 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush": 0, 2023-11-02T08:43:51.138 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_kb": 0, 2023-11-02T08:43:51.139 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict": 0, 2023-11-02T08:43:51.139 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_kb": 0, 2023-11-02T08:43:51.139 INFO:tasks.workunit.client.0.smithi116.stdout: "num_promote": 0, 2023-11-02T08:43:51.139 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_high": 0, 2023-11-02T08:43:51.139 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_low": 0, 2023-11-02T08:43:51.140 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_some": 0, 2023-11-02T08:43:51.140 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_full": 0, 2023-11-02T08:43:51.140 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_pinned": 0, 2023-11-02T08:43:51.140 INFO:tasks.workunit.client.0.smithi116.stdout: "num_legacy_snapsets": 0, 2023-11-02T08:43:51.140 INFO:tasks.workunit.client.0.smithi116.stdout: "num_large_omap_objects": 0, 2023-11-02T08:43:51.140 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_manifest": 0, 2023-11-02T08:43:51.141 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_bytes": 0, 2023-11-02T08:43:51.141 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_keys": 0, 2023-11-02T08:43:51.141 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_repaired": 0 2023-11-02T08:43:51.141 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.141 INFO:tasks.workunit.client.0.smithi116.stdout: "store_stats": { 2023-11-02T08:43:51.142 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 0, 2023-11-02T08:43:51.142 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 0, 2023-11-02T08:43:51.142 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:43:51.142 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 0, 2023-11-02T08:43:51.142 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 0, 2023-11-02T08:43:51.143 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:43:51.143 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:43:51.143 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:43:51.143 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 0, 2023-11-02T08:43:51.143 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 0 2023-11-02T08:43:51.143 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.144 INFO:tasks.workunit.client.0.smithi116.stdout: "log_size": 420, 2023-11-02T08:43:51.144 INFO:tasks.workunit.client.0.smithi116.stdout: "ondisk_log_size": 420, 2023-11-02T08:43:51.144 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 64, 2023-11-02T08:43:51.144 INFO:tasks.workunit.client.0.smithi116.stdout: "acting": 64, 2023-11-02T08:43:51.144 INFO:tasks.workunit.client.0.smithi116.stdout: "num_store_stats": 0 2023-11-02T08:43:51.145 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.145 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_sum": { 2023-11-02T08:43:51.145 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 0, 2023-11-02T08:43:51.145 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 0, 2023-11-02T08:43:51.145 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pgs": 96, 2023-11-02T08:43:51.145 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osds": 3, 2023-11-02T08:43:51.146 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_osds": 3, 2023-11-02T08:43:51.146 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_omap_osds": 3, 2023-11-02T08:43:51.146 INFO:tasks.workunit.client.0.smithi116.stdout: "kb": 314572800, 2023-11-02T08:43:51.146 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used": 84280, 2023-11-02T08:43:51.146 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_data": 2592, 2023-11-02T08:43:51.147 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_omap": 2, 2023-11-02T08:43:51.147 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_meta": 81661, 2023-11-02T08:43:51.147 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_avail": 314488520, 2023-11-02T08:43:51.147 INFO:tasks.workunit.client.0.smithi116.stdout: "statfs": { 2023-11-02T08:43:51.147 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 322122547200, 2023-11-02T08:43:51.147 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 322036244480, 2023-11-02T08:43:51.148 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:43:51.148 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 2654208, 2023-11-02T08:43:51.148 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 608014, 2023-11-02T08:43:51.148 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:43:51.149 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:43:51.149 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:43:51.149 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 2366, 2023-11-02T08:43:51.149 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 83621570 2023-11-02T08:43:51.149 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.149 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_peers": [], 2023-11-02T08:43:51.150 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_trim_queue_len": 0, 2023-11-02T08:43:51.150 INFO:tasks.workunit.client.0.smithi116.stdout: "num_snap_trimming": 0, 2023-11-02T08:43:51.150 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shards_repaired": 0, 2023-11-02T08:43:51.150 INFO:tasks.workunit.client.0.smithi116.stdout: "op_queue_age_hist": { 2023-11-02T08:43:51.150 INFO:tasks.workunit.client.0.smithi116.stdout: "histogram": [], 2023-11-02T08:43:51.151 INFO:tasks.workunit.client.0.smithi116.stdout: "upper_bound": 1 2023-11-02T08:43:51.151 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.151 INFO:tasks.workunit.client.0.smithi116.stdout: "perf_stat": { 2023-11-02T08:43:51.151 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ms": 143, 2023-11-02T08:43:51.151 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ms": 143, 2023-11-02T08:43:51.152 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ns": 143000000, 2023-11-02T08:43:51.152 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ns": 143000000 2023-11-02T08:43:51.152 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.152 INFO:tasks.workunit.client.0.smithi116.stdout: "alerts": [], 2023-11-02T08:43:51.152 INFO:tasks.workunit.client.0.smithi116.stdout: "network_ping_times": [] 2023-11-02T08:43:51.152 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.153 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_sum_by_class": { 2023-11-02T08:43:51.153 INFO:tasks.workunit.client.0.smithi116.stdout: "hdd": { 2023-11-02T08:43:51.153 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 0, 2023-11-02T08:43:51.153 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 0, 2023-11-02T08:43:51.153 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pgs": 96, 2023-11-02T08:43:51.153 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osds": 3, 2023-11-02T08:43:51.154 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_osds": 3, 2023-11-02T08:43:51.154 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_omap_osds": 3, 2023-11-02T08:43:51.154 INFO:tasks.workunit.client.0.smithi116.stdout: "kb": 314572800, 2023-11-02T08:43:51.154 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used": 84280, 2023-11-02T08:43:51.154 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_data": 2592, 2023-11-02T08:43:51.155 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_omap": 2, 2023-11-02T08:43:51.155 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_meta": 81661, 2023-11-02T08:43:51.155 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_avail": 314488520, 2023-11-02T08:43:51.155 INFO:tasks.workunit.client.0.smithi116.stdout: "statfs": { 2023-11-02T08:43:51.155 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 322122547200, 2023-11-02T08:43:51.155 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 322036244480, 2023-11-02T08:43:51.156 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:43:51.156 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 2654208, 2023-11-02T08:43:51.156 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 608014, 2023-11-02T08:43:51.156 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:43:51.156 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:43:51.157 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:43:51.157 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 2366, 2023-11-02T08:43:51.157 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 83621570 2023-11-02T08:43:51.157 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.157 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_peers": [], 2023-11-02T08:43:51.157 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_trim_queue_len": 0, 2023-11-02T08:43:51.158 INFO:tasks.workunit.client.0.smithi116.stdout: "num_snap_trimming": 0, 2023-11-02T08:43:51.158 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shards_repaired": 0, 2023-11-02T08:43:51.158 INFO:tasks.workunit.client.0.smithi116.stdout: "op_queue_age_hist": { 2023-11-02T08:43:51.158 INFO:tasks.workunit.client.0.smithi116.stdout: "histogram": [], 2023-11-02T08:43:51.158 INFO:tasks.workunit.client.0.smithi116.stdout: "upper_bound": 1 2023-11-02T08:43:51.159 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.159 INFO:tasks.workunit.client.0.smithi116.stdout: "perf_stat": { 2023-11-02T08:43:51.159 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ms": 143, 2023-11-02T08:43:51.159 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ms": 143, 2023-11-02T08:43:51.159 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ns": 143000000, 2023-11-02T08:43:51.159 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ns": 143000000 2023-11-02T08:43:51.160 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.160 INFO:tasks.workunit.client.0.smithi116.stdout: "alerts": [], 2023-11-02T08:43:51.160 INFO:tasks.workunit.client.0.smithi116.stdout: "network_ping_times": [] 2023-11-02T08:43:51.160 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.160 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.161 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_stats": [ 2023-11-02T08:43:51.161 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.161 INFO:tasks.workunit.client.0.smithi116.stdout: "poolid": 1, 2023-11-02T08:43:51.161 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg": 32, 2023-11-02T08:43:51.161 INFO:tasks.workunit.client.0.smithi116.stdout: "stat_sum": { 2023-11-02T08:43:51.161 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes": 46451, 2023-11-02T08:43:51.162 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects": 96, 2023-11-02T08:43:51.162 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_clones": 0, 2023-11-02T08:43:51.162 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_copies": 288, 2023-11-02T08:43:51.162 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing_on_primary": 0, 2023-11-02T08:43:51.162 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing": 0, 2023-11-02T08:43:51.163 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_degraded": 96, 2023-11-02T08:43:51.163 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_misplaced": 0, 2023-11-02T08:43:51.163 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_unfound": 0, 2023-11-02T08:43:51.163 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_dirty": 96, 2023-11-02T08:43:51.163 INFO:tasks.workunit.client.0.smithi116.stdout: "num_whiteouts": 0, 2023-11-02T08:43:51.164 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read": 0, 2023-11-02T08:43:51.164 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read_kb": 0, 2023-11-02T08:43:51.164 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write": 91, 2023-11-02T08:43:51.164 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write_kb": 91, 2023-11-02T08:43:51.164 INFO:tasks.workunit.client.0.smithi116.stdout: "num_scrub_errors": 0, 2023-11-02T08:43:51.164 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shallow_scrub_errors": 0, 2023-11-02T08:43:51.165 INFO:tasks.workunit.client.0.smithi116.stdout: "num_deep_scrub_errors": 0, 2023-11-02T08:43:51.165 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_recovered": 0, 2023-11-02T08:43:51.165 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_recovered": 0, 2023-11-02T08:43:51.165 INFO:tasks.workunit.client.0.smithi116.stdout: "num_keys_recovered": 0, 2023-11-02T08:43:51.165 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_omap": 0, 2023-11-02T08:43:51.166 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_hit_set_archive": 0, 2023-11-02T08:43:51.166 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_hit_set_archive": 0, 2023-11-02T08:43:51.166 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush": 0, 2023-11-02T08:43:51.166 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_kb": 0, 2023-11-02T08:43:51.166 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict": 0, 2023-11-02T08:43:51.167 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_kb": 0, 2023-11-02T08:43:51.167 INFO:tasks.workunit.client.0.smithi116.stdout: "num_promote": 0, 2023-11-02T08:43:51.167 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_high": 0, 2023-11-02T08:43:51.167 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_low": 0, 2023-11-02T08:43:51.167 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_some": 0, 2023-11-02T08:43:51.167 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_full": 0, 2023-11-02T08:43:51.168 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_pinned": 0, 2023-11-02T08:43:51.168 INFO:tasks.workunit.client.0.smithi116.stdout: "num_legacy_snapsets": 0, 2023-11-02T08:43:51.168 INFO:tasks.workunit.client.0.smithi116.stdout: "num_large_omap_objects": 0, 2023-11-02T08:43:51.168 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_manifest": 0, 2023-11-02T08:43:51.168 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_bytes": 0, 2023-11-02T08:43:51.169 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_keys": 0, 2023-11-02T08:43:51.169 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_repaired": 0 2023-11-02T08:43:51.169 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.169 INFO:tasks.workunit.client.0.smithi116.stdout: "store_stats": { 2023-11-02T08:43:51.169 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 0, 2023-11-02T08:43:51.169 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 0, 2023-11-02T08:43:51.170 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:43:51.170 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 1220608, 2023-11-02T08:43:51.170 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 142522, 2023-11-02T08:43:51.170 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:43:51.170 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:43:51.171 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:43:51.171 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 0, 2023-11-02T08:43:51.171 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 0 2023-11-02T08:43:51.171 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.171 INFO:tasks.workunit.client.0.smithi116.stdout: "log_size": 420, 2023-11-02T08:43:51.171 INFO:tasks.workunit.client.0.smithi116.stdout: "ondisk_log_size": 420, 2023-11-02T08:43:51.172 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 64, 2023-11-02T08:43:51.172 INFO:tasks.workunit.client.0.smithi116.stdout: "acting": 64, 2023-11-02T08:43:51.172 INFO:tasks.workunit.client.0.smithi116.stdout: "num_store_stats": 3 2023-11-02T08:43:51.172 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.172 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:51.173 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_stats": [ 2023-11-02T08:43:51.173 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.173 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:43:51.173 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 231928234004 2023-11-02T08:43:51.173 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.173 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.174 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:43:51.174 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 42949672988 2023-11-02T08:43:51.174 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.174 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.174 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:43:51.174 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 249108103187 2023-11-02T08:43:51.175 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.175 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:51.175 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_by_state": [ 2023-11-02T08:43:51.175 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.175 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "active+undersized+degraded", 2023-11-02T08:43:51.176 INFO:tasks.workunit.client.0.smithi116.stdout: "num": 32 2023-11-02T08:43:51.176 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.176 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:51.176 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_by_osd": [ 2023-11-02T08:43:51.176 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.176 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:43:51.177 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 18, 2023-11-02T08:43:51.177 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 32, 2023-11-02T08:43:51.177 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:43:51.177 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.177 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.178 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:43:51.178 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 14, 2023-11-02T08:43:51.178 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 32, 2023-11-02T08:43:51.178 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:43:51.178 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.178 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.179 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:43:51.179 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 0, 2023-11-02T08:43:51.179 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 0, 2023-11-02T08:43:51.179 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:43:51.179 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.180 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:51.180 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snaps": [ 2023-11-02T08:43:51.180 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:43:51.180 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": 1, 2023-11-02T08:43:51.180 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snaps": {} 2023-11-02T08:43:51.180 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.181 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:43:51.181 INFO:tasks.workunit.client.0.smithi116.stdout: "servicemap": { 2023-11-02T08:43:51.181 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 6, 2023-11-02T08:43:51.181 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:43:34.842567+0000", 2023-11-02T08:43:51.181 INFO:tasks.workunit.client.0.smithi116.stdout: "services": {} 2023-11-02T08:43:51.181 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:43:51.182 INFO:tasks.workunit.client.0.smithi116.stdout: "mgrstat_first_committed": 1, 2023-11-02T08:43:51.182 INFO:tasks.workunit.client.0.smithi116.stdout: "mgrstat_last_committed": 105, 2023-11-02T08:43:51.182 INFO:tasks.workunit.client.0.smithi116.stdout: "logm_first_committed": 1, 2023-11-02T08:43:51.182 INFO:tasks.workunit.client.0.smithi116.stdout: "logm_last_committed": 459, 2023-11-02T08:43:51.182 INFO:tasks.workunit.client.0.smithi116.stdout: "paxos": { 2023-11-02T08:43:51.183 INFO:tasks.workunit.client.0.smithi116.stdout: "first_committed": 1, 2023-11-02T08:43:51.183 INFO:tasks.workunit.client.0.smithi116.stdout: "last_committed": 664, 2023-11-02T08:43:51.183 INFO:tasks.workunit.client.0.smithi116.stdout: "last_pn": 0, 2023-11-02T08:43:51.183 INFO:tasks.workunit.client.0.smithi116.stdout: "accepted_pn": 0 2023-11-02T08:43:51.183 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:43:51.183 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T08:43:51.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2023-11-02T08:43:51.184 INFO:tasks.workunit.client.0.smithi116.stdout:delay recovery 2023-11-02T08:43:51.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: echo 'delay recovery' 2023-11-02T08:43:51.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph pg dump pgs 2023-11-02T08:43:51.322 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:43:51.322 INFO:tasks.workunit.client.0.smithi116.stdout:1.1f 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945032+0000 32'97 66:129 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.322 INFO:tasks.workunit.client.0.smithi116.stdout:1.1e 4 0 4 0 0 1947 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.946255+0000 34'98 66:153 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.322 INFO:tasks.workunit.client.0.smithi116.stdout:1.1d 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944739+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.323 INFO:tasks.workunit.client.0.smithi116.stdout:1.b 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920318+0000 32'97 66:132 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.323 INFO:tasks.workunit.client.0.smithi116.stdout:1.a 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944603+0000 32'97 66:120 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.323 INFO:tasks.workunit.client.0.smithi116.stdout:1.9 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920339+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.323 INFO:tasks.workunit.client.0.smithi116.stdout:1.8 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945695+0000 32'97 66:120 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.324 INFO:tasks.workunit.client.0.smithi116.stdout:1.7 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.851862+0000 32'97 66:129 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.324 INFO:tasks.workunit.client.0.smithi116.stdout:1.6 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.946582+0000 32'97 66:141 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.324 INFO:tasks.workunit.client.0.smithi116.stdout:1.5 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945458+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.324 INFO:tasks.workunit.client.0.smithi116.stdout:1.4 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944062+0000 32'97 66:132 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.324 INFO:tasks.workunit.client.0.smithi116.stdout:1.3 4 0 4 0 0 1948 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.947103+0000 34'98 66:141 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.325 INFO:tasks.workunit.client.0.smithi116.stdout:1.2 1 0 1 0 0 483 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944339+0000 32'97 66:153 [0,2] 0 [0,2] 0 32'97 2023-11-02T08:42:12.415700+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T18:58:39.256358+0000 1 0 2023-11-02T08:43:51.325 INFO:tasks.workunit.client.0.smithi116.stdout:1.c 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920311+0000 32'97 66:132 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.325 INFO:tasks.workunit.client.0.smithi116.stdout:1.1 2 0 2 0 0 966 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944180+0000 32'97 66:143 [2,0] 2 [2,0] 2 32'97 2023-11-02T08:42:09.981464+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T14:45:22.323682+0000 2 0 2023-11-02T08:43:51.325 INFO:tasks.workunit.client.0.smithi116.stdout:1.1c 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944438+0000 32'97 66:132 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.325 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.947058+0000 32'97 66:132 [0,2] 0 [0,2] 0 32'97 2023-11-02T08:42:07.198292+0000 32'97 2023-11-02T08:42:07.198292+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.326 INFO:tasks.workunit.client.0.smithi116.stdout:1.d 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.946518+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.326 INFO:tasks.workunit.client.0.smithi116.stdout:1.e 3 0 3 0 0 1111 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.945332+0000 66'98 66:142 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.326 INFO:tasks.workunit.client.0.smithi116.stdout:1.f 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.946682+0000 32'97 66:129 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.326 INFO:tasks.workunit.client.0.smithi116.stdout:1.10 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944393+0000 32'97 66:132 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.327 INFO:tasks.workunit.client.0.smithi116.stdout:1.11 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944108+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.327 INFO:tasks.workunit.client.0.smithi116.stdout:1.12 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944113+0000 32'97 66:141 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.327 INFO:tasks.workunit.client.0.smithi116.stdout:1.13 3 0 3 0 0 1449 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920751+0000 32'97 66:141 [2,0] 2 [2,0] 2 32'97 2023-11-02T08:42:10.923269+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T08:52:50.221029+0000 3 0 2023-11-02T08:43:51.327 INFO:tasks.workunit.client.0.smithi116.stdout:1.14 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.947335+0000 32'97 66:132 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.327 INFO:tasks.workunit.client.0.smithi116.stdout:1.15 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.947192+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.328 INFO:tasks.workunit.client.0.smithi116.stdout:1.16 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944598+0000 32'97 66:141 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.328 INFO:tasks.workunit.client.0.smithi116.stdout:1.17 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945003+0000 32'97 66:129 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.328 INFO:tasks.workunit.client.0.smithi116.stdout:1.18 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944425+0000 32'97 66:120 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.328 INFO:tasks.workunit.client.0.smithi116.stdout:1.19 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944351+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.328 INFO:tasks.workunit.client.0.smithi116.stdout:1.1a 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920724+0000 32'97 66:120 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.329 INFO:tasks.workunit.client.0.smithi116.stdout:1.1b 4 0 4 0 0 1947 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.920740+0000 34'98 66:143 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.329 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:43:51.329 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:43:51.329 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:43:51.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2023-11-02T08:43:51.333 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.0 2023-11-02T08:43:51.333 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-11-02T08:43:51.333 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-11-02T08:43:51.334 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:43:51.334 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:43:51.334 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:43:51.334 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.0.asok 2023-11-02T08:43:51.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2023-11-02T08:43:51.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.0.asok set_recovery_delay 100000 2023-11-02T08:43:51.412 INFO:tasks.workunit.client.0.smithi116.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2023-11-02T08:43:51.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2023-11-02T08:43:51.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.2 2023-11-02T08:43:51.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-11-02T08:43:51.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-11-02T08:43:51.421 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:43:51.421 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:43:51.422 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:43:51.422 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.2.asok 2023-11-02T08:43:51.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2023-11-02T08:43:51.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.2.asok set_recovery_delay 100000 2023-11-02T08:43:51.501 INFO:tasks.workunit.client.0.smithi116.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2023-11-02T08:43:51.508 INFO:tasks.workunit.client.0.smithi116.stdout:revive divergent 1 2023-11-02T08:43:51.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:769: TEST_divergent_3: echo 'revive divergent 1' 2023-11-02T08:43:51.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2023-11-02T08:43:51.872 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:43:51.872 INFO:tasks.workunit.client.0.smithi116.stdout:1.1f 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945032+0000 32'97 66:129 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.872 INFO:tasks.workunit.client.0.smithi116.stdout:1.1e 4 0 4 0 0 1947 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.946255+0000 34'98 66:153 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.872 INFO:tasks.workunit.client.0.smithi116.stdout:1.1d 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944739+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.873 INFO:tasks.workunit.client.0.smithi116.stdout:1.b 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920318+0000 32'97 66:132 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.873 INFO:tasks.workunit.client.0.smithi116.stdout:1.a 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944603+0000 32'97 66:120 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.873 INFO:tasks.workunit.client.0.smithi116.stdout:1.9 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920339+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.873 INFO:tasks.workunit.client.0.smithi116.stdout:1.8 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945695+0000 32'97 66:120 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.873 INFO:tasks.workunit.client.0.smithi116.stdout:1.7 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.851862+0000 32'97 66:129 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.874 INFO:tasks.workunit.client.0.smithi116.stdout:1.6 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.946582+0000 32'97 66:141 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.874 INFO:tasks.workunit.client.0.smithi116.stdout:1.5 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945458+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.874 INFO:tasks.workunit.client.0.smithi116.stdout:1.4 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944062+0000 32'97 66:132 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.874 INFO:tasks.workunit.client.0.smithi116.stdout:1.3 4 0 4 0 0 1948 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.947103+0000 34'98 66:141 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.875 INFO:tasks.workunit.client.0.smithi116.stdout:1.2 1 0 1 0 0 483 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944339+0000 32'97 66:153 [0,2] 0 [0,2] 0 32'97 2023-11-02T08:42:12.415700+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T18:58:39.256358+0000 1 0 2023-11-02T08:43:51.875 INFO:tasks.workunit.client.0.smithi116.stdout:1.c 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920311+0000 32'97 66:132 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.875 INFO:tasks.workunit.client.0.smithi116.stdout:1.1 2 0 2 0 0 966 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944180+0000 32'97 66:143 [2,0] 2 [2,0] 2 32'97 2023-11-02T08:42:09.981464+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T14:45:22.323682+0000 2 0 2023-11-02T08:43:51.875 INFO:tasks.workunit.client.0.smithi116.stdout:1.1c 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944438+0000 32'97 66:132 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.875 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.947058+0000 32'97 66:132 [0,2] 0 [0,2] 0 32'97 2023-11-02T08:42:07.198292+0000 32'97 2023-11-02T08:42:07.198292+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.876 INFO:tasks.workunit.client.0.smithi116.stdout:1.d 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.946518+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.876 INFO:tasks.workunit.client.0.smithi116.stdout:1.e 3 0 3 0 0 1111 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.945332+0000 66'98 66:142 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.876 INFO:tasks.workunit.client.0.smithi116.stdout:1.f 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.946682+0000 32'97 66:129 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.876 INFO:tasks.workunit.client.0.smithi116.stdout:1.10 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944393+0000 32'97 66:132 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.876 INFO:tasks.workunit.client.0.smithi116.stdout:1.11 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944108+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.877 INFO:tasks.workunit.client.0.smithi116.stdout:1.12 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944113+0000 32'97 66:141 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.877 INFO:tasks.workunit.client.0.smithi116.stdout:1.13 3 0 3 0 0 1449 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920751+0000 32'97 66:141 [2,0] 2 [2,0] 2 32'97 2023-11-02T08:42:10.923269+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T08:52:50.221029+0000 3 0 2023-11-02T08:43:51.877 INFO:tasks.workunit.client.0.smithi116.stdout:1.14 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.947335+0000 32'97 66:132 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.877 INFO:tasks.workunit.client.0.smithi116.stdout:1.15 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.947192+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.877 INFO:tasks.workunit.client.0.smithi116.stdout:1.16 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944598+0000 32'97 66:141 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.877 INFO:tasks.workunit.client.0.smithi116.stdout:1.17 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945003+0000 32'97 66:129 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.878 INFO:tasks.workunit.client.0.smithi116.stdout:1.18 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944425+0000 32'97 66:120 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.878 INFO:tasks.workunit.client.0.smithi116.stdout:1.19 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944351+0000 32'97 66:132 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.878 INFO:tasks.workunit.client.0.smithi116.stdout:1.1a 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920724+0000 32'97 66:120 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.878 INFO:tasks.workunit.client.0.smithi116.stdout:1.1b 4 0 4 0 0 1947 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.920740+0000 34'98 66:143 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:51.878 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:43:51.879 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:43:51.879 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:43:51.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: ceph osd set noup 2023-11-02T08:43:52.643 INFO:tasks.workunit.client.0.smithi116.stderr:noup is set 2023-11-02T08:43:52.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: activate_osd td/divergent-priors 1 2023-11-02T08:43:52.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2023-11-02T08:43:52.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:43:52.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2023-11-02T08:43:52.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:43:52.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2023-11-02T08:43:52.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:43:52.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:43:52.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:43:52.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:43:52.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2023-11-02T08:43:52.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2023-11-02T08:43:52.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:43:52.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:43:52.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:43:52.660 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:43:52.661 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:43:52.661 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:43:52.661 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:43:52.661 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:43:52.661 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:43:52.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:43:52.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:43:52.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:43:52.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:43:52.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:43:52.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:43:52.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:43:52.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:43:52.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:43:52.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:43:52.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:43:52.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2023-11-02T08:43:52.665 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T08:43:52.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2023-11-02T08:43:52.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:43:52.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2023-11-02T08:43:52.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2023-11-02T08:43:52.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:43:52.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:43:52.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:43:52.690 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:43:52.687+0000 7f74e88c9500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:43:52.692 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:43:52.689+0000 7f74e88c9500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:43:52.693 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:43:52.691+0000 7f74e88c9500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:43:53.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2023-11-02T08:43:53.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: sleep 5 2023-11-02T08:43:53.215 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:43:53.212+0000 7f74e88c9500 -1 Falling back to public interface 2023-11-02T08:43:56.027 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:43:56.024+0000 7f74e88c9500 -1 osd.1 44 log_to_monitors true 2023-11-02T08:43:58.067 INFO:tasks.workunit.client.0.smithi116.stdout:delay recovery divergent 2023-11-02T08:43:58.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:775: TEST_divergent_3: echo 'delay recovery divergent' 2023-11-02T08:43:58.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:776: TEST_divergent_3: ceph pg dump pgs 2023-11-02T08:43:58.437 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:43:58.437 INFO:tasks.workunit.client.0.smithi116.stdout:1.1f 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945032+0000 32'97 67:131 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.438 INFO:tasks.workunit.client.0.smithi116.stdout:1.1e 4 0 4 0 0 1947 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.946255+0000 34'98 66:154 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.438 INFO:tasks.workunit.client.0.smithi116.stdout:1.1d 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944739+0000 32'97 67:134 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.438 INFO:tasks.workunit.client.0.smithi116.stdout:1.b 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920318+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.438 INFO:tasks.workunit.client.0.smithi116.stdout:1.a 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944603+0000 32'97 67:122 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.439 INFO:tasks.workunit.client.0.smithi116.stdout:1.9 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920339+0000 32'97 67:134 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.439 INFO:tasks.workunit.client.0.smithi116.stdout:1.8 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945695+0000 32'97 67:122 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.439 INFO:tasks.workunit.client.0.smithi116.stdout:1.7 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.851862+0000 32'97 67:131 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.439 INFO:tasks.workunit.client.0.smithi116.stdout:1.6 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.946582+0000 32'97 66:142 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.439 INFO:tasks.workunit.client.0.smithi116.stdout:1.5 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945458+0000 32'97 67:134 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.440 INFO:tasks.workunit.client.0.smithi116.stdout:1.4 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944062+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.440 INFO:tasks.workunit.client.0.smithi116.stdout:1.3 4 0 4 0 0 1948 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.947103+0000 34'98 67:143 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.440 INFO:tasks.workunit.client.0.smithi116.stdout:1.2 1 0 1 0 0 483 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944339+0000 32'97 66:154 [0,2] 0 [0,2] 0 32'97 2023-11-02T08:42:12.415700+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T18:58:39.256358+0000 1 0 2023-11-02T08:43:58.440 INFO:tasks.workunit.client.0.smithi116.stdout:1.c 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920311+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.440 INFO:tasks.workunit.client.0.smithi116.stdout:1.1 2 0 2 0 0 966 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944180+0000 32'97 67:145 [2,0] 2 [2,0] 2 32'97 2023-11-02T08:42:09.981464+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T14:45:22.323682+0000 2 0 2023-11-02T08:43:58.441 INFO:tasks.workunit.client.0.smithi116.stdout:1.1c 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944438+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.441 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.947058+0000 32'97 66:133 [0,2] 0 [0,2] 0 32'97 2023-11-02T08:42:07.198292+0000 32'97 2023-11-02T08:42:07.198292+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.441 INFO:tasks.workunit.client.0.smithi116.stdout:1.d 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.946518+0000 32'97 67:134 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.441 INFO:tasks.workunit.client.0.smithi116.stdout:1.e 3 0 3 0 0 1111 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.945332+0000 66'98 66:143 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.441 INFO:tasks.workunit.client.0.smithi116.stdout:1.f 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.946682+0000 32'97 67:131 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.441 INFO:tasks.workunit.client.0.smithi116.stdout:1.10 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944393+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.442 INFO:tasks.workunit.client.0.smithi116.stdout:1.11 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944108+0000 32'97 67:134 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.442 INFO:tasks.workunit.client.0.smithi116.stdout:1.12 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944113+0000 32'97 66:142 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.442 INFO:tasks.workunit.client.0.smithi116.stdout:1.13 3 0 3 0 0 1449 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920751+0000 32'97 67:143 [2,0] 2 [2,0] 2 32'97 2023-11-02T08:42:10.923269+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T08:52:50.221029+0000 3 0 2023-11-02T08:43:58.442 INFO:tasks.workunit.client.0.smithi116.stdout:1.14 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.947335+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.442 INFO:tasks.workunit.client.0.smithi116.stdout:1.15 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.947192+0000 32'97 67:134 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.443 INFO:tasks.workunit.client.0.smithi116.stdout:1.16 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944598+0000 32'97 66:142 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.443 INFO:tasks.workunit.client.0.smithi116.stdout:1.17 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945003+0000 32'97 67:131 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.443 INFO:tasks.workunit.client.0.smithi116.stdout:1.18 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944425+0000 32'97 67:122 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.443 INFO:tasks.workunit.client.0.smithi116.stdout:1.19 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944351+0000 32'97 67:134 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.443 INFO:tasks.workunit.client.0.smithi116.stdout:1.1a 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920724+0000 32'97 67:122 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.444 INFO:tasks.workunit.client.0.smithi116.stdout:1.1b 4 0 4 0 0 1947 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.920740+0000 34'98 66:144 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:43:58.444 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:43:58.444 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:43:58.444 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:43:58.448 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: get_asok_path osd.1 2023-11-02T08:43:58.448 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-11-02T08:43:58.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-11-02T08:43:58.449 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:43:58.449 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:43:58.449 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:43:58.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.1.asok 2023-11-02T08:43:58.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: CEPH_ARGS= 2023-11-02T08:43:58.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.1.asok set_recovery_delay 100000 2023-11-02T08:43:58.529 INFO:tasks.workunit.client.0.smithi116.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2023-11-02T08:43:58.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: ceph osd unset noup 2023-11-02T08:43:59.164 INFO:tasks.workunit.client.0.smithi116.stderr:noup is unset 2023-11-02T08:43:59.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:781: TEST_divergent_3: wait_for_osd up 0 2023-11-02T08:43:59.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:43:59.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:43:59.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:43:59.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:43:59.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:43:59.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:43:59.177 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:43:59.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:43:59.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:43:59.584 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 54 up_thru 63 down_at 45 last_clean_interval [5,38) [v2:127.0.0.1:6802/120543,v1:127.0.0.1:6803/120543] [v2:127.0.0.1:6804/120543,v1:127.0.0.1:6805/120543] exists,up 584ee92b-5cfd-43f0-8535-9a70b0612348 2023-11-02T08:43:59.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:43:59.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:43:59.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:43:59.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: wait_for_osd up 1 2023-11-02T08:43:59.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:43:59.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:43:59.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:43:59.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:43:59.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:43:59.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:43:59.588 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:43:59.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:43:59.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:44:00.131 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 71 up_thru 71 down_at 46 last_clean_interval [10,44) [v2:127.0.0.1:6818/123641,v1:127.0.0.1:6819/123641] [v2:127.0.0.1:6820/123641,v1:127.0.0.1:6821/123641] exists,up 4d681ad5-42d6-443f-ac67-86739e3ca916 2023-11-02T08:44:00.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:44:00.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:44:00.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:44:00.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: wait_for_osd up 2 2023-11-02T08:44:00.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:44:00.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T08:44:00.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:44:00.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:44:00.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:44:00.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:44:00.135 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:44:00.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:44:00.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:44:00.600 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 58 up_thru 71 down_at 48 last_clean_interval [15,39) [v2:127.0.0.1:6810/121087,v1:127.0.0.1:6811/121087] [v2:127.0.0.1:6812/121087,v1:127.0.0.1:6813/121087] exists,up 8b4576fe-bdf3-4241-80eb-7d5e8439887c 2023-11-02T08:44:00.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:44:00.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:44:00.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:44:00.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:785: TEST_divergent_3: ceph pg dump pgs 2023-11-02T08:44:00.956 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:44:00.956 INFO:tasks.workunit.client.0.smithi116.stdout:1.1f 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945032+0000 32'97 67:131 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.957 INFO:tasks.workunit.client.0.smithi116.stdout:1.1e 4 0 4 0 0 1947 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.946255+0000 34'98 66:154 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.957 INFO:tasks.workunit.client.0.smithi116.stdout:1.1d 3 0 0 0 0 1464 0 0 13 84 13 peering 2023-11-02T08:43:59.237921+0000 32'97 71:138 [2,1,0] 2 [2,1,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.957 INFO:tasks.workunit.client.0.smithi116.stdout:1.b 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920318+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.957 INFO:tasks.workunit.client.0.smithi116.stdout:1.a 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944603+0000 32'97 67:122 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.957 INFO:tasks.workunit.client.0.smithi116.stdout:1.9 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920339+0000 32'97 67:134 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.958 INFO:tasks.workunit.client.0.smithi116.stdout:1.8 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945695+0000 32'97 67:122 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.958 INFO:tasks.workunit.client.0.smithi116.stdout:1.7 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.851862+0000 32'97 67:131 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.958 INFO:tasks.workunit.client.0.smithi116.stdout:1.6 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.946582+0000 32'97 66:142 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.958 INFO:tasks.workunit.client.0.smithi116.stdout:1.5 3 0 0 0 0 1464 0 0 13 84 13 peering 2023-11-02T08:43:59.230680+0000 32'97 71:138 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.958 INFO:tasks.workunit.client.0.smithi116.stdout:1.4 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944062+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.959 INFO:tasks.workunit.client.0.smithi116.stdout:1.3 4 0 4 0 0 1948 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.947103+0000 34'98 67:143 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.959 INFO:tasks.workunit.client.0.smithi116.stdout:1.2 1 0 0 0 0 483 0 0 13 84 13 active+clean 2023-11-02T08:44:00.276808+0000 32'97 73:166 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:42:12.415700+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T14:54:52.696258+0000 1 0 2023-11-02T08:44:00.959 INFO:tasks.workunit.client.0.smithi116.stdout:1.c 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920311+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.959 INFO:tasks.workunit.client.0.smithi116.stdout:1.1 2 0 0 0 0 966 0 0 13 84 13 peering 2023-11-02T08:43:59.238778+0000 32'97 71:149 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:42:09.981464+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T19:09:24.266368+0000 2 0 2023-11-02T08:44:00.959 INFO:tasks.workunit.client.0.smithi116.stdout:1.1c 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944438+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.960 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.947058+0000 32'97 66:133 [0,2] 0 [0,2] 0 32'97 2023-11-02T08:42:07.198292+0000 32'97 2023-11-02T08:42:07.198292+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.960 INFO:tasks.workunit.client.0.smithi116.stdout:1.d 3 0 0 0 0 1464 0 0 13 84 13 peering 2023-11-02T08:43:59.239974+0000 32'97 71:138 [2,1,0] 2 [2,1,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.960 INFO:tasks.workunit.client.0.smithi116.stdout:1.e 3 0 3 0 0 1111 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.945332+0000 66'98 66:143 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.960 INFO:tasks.workunit.client.0.smithi116.stdout:1.f 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.946682+0000 32'97 67:131 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.960 INFO:tasks.workunit.client.0.smithi116.stdout:1.10 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944393+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.961 INFO:tasks.workunit.client.0.smithi116.stdout:1.11 3 0 0 0 0 1464 0 0 13 84 13 peering 2023-11-02T08:43:59.238170+0000 32'97 71:139 [2,0,1] 2 [2,0,1] 2 39'98 2023-11-02T08:42:18.867158+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.961 INFO:tasks.workunit.client.0.smithi116.stdout:1.12 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:00.276703+0000 32'97 73:154 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:42:15.453129+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.961 INFO:tasks.workunit.client.0.smithi116.stdout:1.13 3 0 3 0 0 1449 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920751+0000 32'97 67:143 [2,0] 2 [2,0] 2 32'97 2023-11-02T08:42:10.923269+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T08:52:50.221029+0000 3 0 2023-11-02T08:44:00.961 INFO:tasks.workunit.client.0.smithi116.stdout:1.14 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.947335+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.961 INFO:tasks.workunit.client.0.smithi116.stdout:1.15 3 0 0 0 0 1464 0 0 13 84 13 peering 2023-11-02T08:43:59.241077+0000 32'97 71:138 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.962 INFO:tasks.workunit.client.0.smithi116.stdout:1.16 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944598+0000 32'97 66:142 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.962 INFO:tasks.workunit.client.0.smithi116.stdout:1.17 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945003+0000 32'97 67:131 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.962 INFO:tasks.workunit.client.0.smithi116.stdout:1.18 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944425+0000 32'97 67:122 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.962 INFO:tasks.workunit.client.0.smithi116.stdout:1.19 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944351+0000 32'97 67:134 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.962 INFO:tasks.workunit.client.0.smithi116.stdout:1.1a 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920724+0000 32'97 67:122 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.962 INFO:tasks.workunit.client.0.smithi116.stdout:1.1b 4 0 4 0 0 1947 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.920740+0000 34'98 66:144 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:00.963 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:44:00.963 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:44:00.963 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:44:00.967 INFO:tasks.workunit.client.0.smithi116.stdout:wait for peering 2023-11-02T08:44:00.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'wait for peering' 2023-11-02T08:44:00.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2023-11-02T08:44:01.344 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:44:01.344 INFO:tasks.workunit.client.0.smithi116.stdout:1.1f 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945032+0000 32'97 67:131 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.344 INFO:tasks.workunit.client.0.smithi116.stdout:1.1e 4 0 4 0 0 1947 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.946255+0000 34'98 66:154 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.345 INFO:tasks.workunit.client.0.smithi116.stdout:1.1d 3 0 0 0 0 1464 0 0 13 84 13 peering 2023-11-02T08:43:59.237921+0000 32'97 71:138 [2,1,0] 2 [2,1,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.345 INFO:tasks.workunit.client.0.smithi116.stdout:1.b 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920318+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.345 INFO:tasks.workunit.client.0.smithi116.stdout:1.a 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944603+0000 32'97 67:122 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.345 INFO:tasks.workunit.client.0.smithi116.stdout:1.9 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920339+0000 32'97 67:134 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.345 INFO:tasks.workunit.client.0.smithi116.stdout:1.8 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945695+0000 32'97 67:122 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.346 INFO:tasks.workunit.client.0.smithi116.stdout:1.7 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.851862+0000 32'97 67:131 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.346 INFO:tasks.workunit.client.0.smithi116.stdout:1.6 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.946582+0000 32'97 66:142 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.346 INFO:tasks.workunit.client.0.smithi116.stdout:1.5 3 0 0 0 0 1464 0 0 13 84 13 peering 2023-11-02T08:43:59.230680+0000 32'97 71:138 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.346 INFO:tasks.workunit.client.0.smithi116.stdout:1.4 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944062+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.346 INFO:tasks.workunit.client.0.smithi116.stdout:1.3 4 0 4 0 0 1948 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.947103+0000 34'98 67:143 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.347 INFO:tasks.workunit.client.0.smithi116.stdout:1.2 1 0 0 0 0 483 0 0 13 84 13 active+clean 2023-11-02T08:44:00.276808+0000 32'97 73:166 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:42:12.415700+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T14:54:52.696258+0000 1 0 2023-11-02T08:44:01.347 INFO:tasks.workunit.client.0.smithi116.stdout:1.c 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920311+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.347 INFO:tasks.workunit.client.0.smithi116.stdout:1.1 2 0 0 0 0 966 0 0 13 84 13 peering 2023-11-02T08:43:59.238778+0000 32'97 71:149 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:42:09.981464+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T19:09:24.266368+0000 2 0 2023-11-02T08:44:01.347 INFO:tasks.workunit.client.0.smithi116.stdout:1.1c 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944438+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.347 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.947058+0000 32'97 66:133 [0,2] 0 [0,2] 0 32'97 2023-11-02T08:42:07.198292+0000 32'97 2023-11-02T08:42:07.198292+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.348 INFO:tasks.workunit.client.0.smithi116.stdout:1.d 3 0 0 0 0 1464 0 0 13 84 13 peering 2023-11-02T08:43:59.239974+0000 32'97 71:138 [2,1,0] 2 [2,1,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.348 INFO:tasks.workunit.client.0.smithi116.stdout:1.e 3 0 3 0 0 1111 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.945332+0000 66'98 66:143 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.348 INFO:tasks.workunit.client.0.smithi116.stdout:1.f 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.946682+0000 32'97 67:131 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.348 INFO:tasks.workunit.client.0.smithi116.stdout:1.10 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944393+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.348 INFO:tasks.workunit.client.0.smithi116.stdout:1.11 3 0 0 0 0 1464 0 0 13 84 13 peering 2023-11-02T08:43:59.238170+0000 32'97 71:139 [2,0,1] 2 [2,0,1] 2 39'98 2023-11-02T08:42:18.867158+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.349 INFO:tasks.workunit.client.0.smithi116.stdout:1.12 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:00.276703+0000 32'97 73:154 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:42:15.453129+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.349 INFO:tasks.workunit.client.0.smithi116.stdout:1.13 3 0 3 0 0 1449 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920751+0000 32'97 67:143 [2,0] 2 [2,0] 2 32'97 2023-11-02T08:42:10.923269+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T08:52:50.221029+0000 3 0 2023-11-02T08:44:01.349 INFO:tasks.workunit.client.0.smithi116.stdout:1.14 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.947335+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.349 INFO:tasks.workunit.client.0.smithi116.stdout:1.15 3 0 0 0 0 1464 0 0 13 84 13 peering 2023-11-02T08:43:59.241077+0000 32'97 71:138 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.349 INFO:tasks.workunit.client.0.smithi116.stdout:1.16 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944598+0000 32'97 66:142 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.350 INFO:tasks.workunit.client.0.smithi116.stdout:1.17 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945003+0000 32'97 67:131 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.350 INFO:tasks.workunit.client.0.smithi116.stdout:1.18 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944425+0000 32'97 67:122 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.350 INFO:tasks.workunit.client.0.smithi116.stdout:1.19 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944351+0000 32'97 67:134 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.350 INFO:tasks.workunit.client.0.smithi116.stdout:1.1a 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920724+0000 32'97 67:122 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.350 INFO:tasks.workunit.client.0.smithi116.stdout:1.1b 4 0 4 0 0 1947 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.920740+0000 34'98 66:144 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.351 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:44:01.351 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:44:01.351 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:44:01.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: rados -p test put foo /etc/fstab 2023-11-02T08:44:01.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: echo 'killing divergent 1' 2023-11-02T08:44:01.476 INFO:tasks.workunit.client.0.smithi116.stdout:killing divergent 1 2023-11-02T08:44:01.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:793: TEST_divergent_3: ceph pg dump pgs 2023-11-02T08:44:01.834 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:44:01.835 INFO:tasks.workunit.client.0.smithi116.stdout:1.1f 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945032+0000 32'97 67:131 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.835 INFO:tasks.workunit.client.0.smithi116.stdout:1.1e 4 0 4 0 0 1947 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.946255+0000 34'98 66:154 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.835 INFO:tasks.workunit.client.0.smithi116.stdout:1.1d 3 0 0 0 0 1464 0 0 13 84 13 peering 2023-11-02T08:43:59.237921+0000 32'97 71:138 [2,1,0] 2 [2,1,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.835 INFO:tasks.workunit.client.0.smithi116.stdout:1.b 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920318+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.835 INFO:tasks.workunit.client.0.smithi116.stdout:1.a 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944603+0000 32'97 67:122 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.836 INFO:tasks.workunit.client.0.smithi116.stdout:1.9 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920339+0000 32'97 67:134 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.836 INFO:tasks.workunit.client.0.smithi116.stdout:1.8 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945695+0000 32'97 67:122 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.836 INFO:tasks.workunit.client.0.smithi116.stdout:1.7 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.851862+0000 32'97 67:131 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.836 INFO:tasks.workunit.client.0.smithi116.stdout:1.6 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.946582+0000 32'97 66:142 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.836 INFO:tasks.workunit.client.0.smithi116.stdout:1.5 3 0 0 0 0 1464 0 0 13 84 13 peering 2023-11-02T08:43:59.230680+0000 32'97 71:138 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.837 INFO:tasks.workunit.client.0.smithi116.stdout:1.4 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944062+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.837 INFO:tasks.workunit.client.0.smithi116.stdout:1.3 4 0 4 0 0 1948 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.947103+0000 34'98 67:143 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.837 INFO:tasks.workunit.client.0.smithi116.stdout:1.2 1 0 0 0 0 483 0 0 13 84 13 active+clean 2023-11-02T08:44:00.276808+0000 32'97 73:166 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:42:12.415700+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T14:54:52.696258+0000 1 0 2023-11-02T08:44:01.837 INFO:tasks.workunit.client.0.smithi116.stdout:1.c 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920311+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.837 INFO:tasks.workunit.client.0.smithi116.stdout:1.1 2 0 0 0 0 966 0 0 13 84 13 peering 2023-11-02T08:43:59.238778+0000 32'97 71:149 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:42:09.981464+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T19:09:24.266368+0000 2 0 2023-11-02T08:44:01.838 INFO:tasks.workunit.client.0.smithi116.stdout:1.1c 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944438+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.838 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.947058+0000 32'97 66:133 [0,2] 0 [0,2] 0 32'97 2023-11-02T08:42:07.198292+0000 32'97 2023-11-02T08:42:07.198292+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.838 INFO:tasks.workunit.client.0.smithi116.stdout:1.d 3 0 0 0 0 1464 0 0 13 84 13 peering 2023-11-02T08:43:59.239974+0000 32'97 71:138 [2,1,0] 2 [2,1,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.838 INFO:tasks.workunit.client.0.smithi116.stdout:1.e 3 0 3 0 0 1111 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.945332+0000 66'98 66:143 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.838 INFO:tasks.workunit.client.0.smithi116.stdout:1.f 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.946682+0000 32'97 67:131 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.839 INFO:tasks.workunit.client.0.smithi116.stdout:1.10 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944393+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.839 INFO:tasks.workunit.client.0.smithi116.stdout:1.11 3 0 0 0 0 1464 0 0 13 84 13 peering 2023-11-02T08:43:59.238170+0000 32'97 71:139 [2,0,1] 2 [2,0,1] 2 39'98 2023-11-02T08:42:18.867158+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.839 INFO:tasks.workunit.client.0.smithi116.stdout:1.12 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:00.276703+0000 32'97 73:154 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:42:15.453129+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.839 INFO:tasks.workunit.client.0.smithi116.stdout:1.13 3 0 3 0 0 1449 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920751+0000 32'97 67:143 [2,0] 2 [2,0] 2 32'97 2023-11-02T08:42:10.923269+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T08:52:50.221029+0000 3 0 2023-11-02T08:44:01.839 INFO:tasks.workunit.client.0.smithi116.stdout:1.14 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.947335+0000 32'97 66:133 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.840 INFO:tasks.workunit.client.0.smithi116.stdout:1.15 3 0 0 0 0 1464 0 0 13 84 13 peering 2023-11-02T08:43:59.241077+0000 32'97 71:138 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.840 INFO:tasks.workunit.client.0.smithi116.stdout:1.16 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944598+0000 32'97 66:142 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.840 INFO:tasks.workunit.client.0.smithi116.stdout:1.17 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.945003+0000 32'97 67:131 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.840 INFO:tasks.workunit.client.0.smithi116.stdout:1.18 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944425+0000 32'97 67:122 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.840 INFO:tasks.workunit.client.0.smithi116.stdout:1.19 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.944351+0000 32'97 67:134 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.841 INFO:tasks.workunit.client.0.smithi116.stdout:1.1a 3 0 3 0 0 1464 0 0 13 84 13 active+undersized+degraded 2023-11-02T08:42:52.920724+0000 32'97 67:122 [2,0] 2 [2,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.841 INFO:tasks.workunit.client.0.smithi116.stdout:1.1b 4 0 4 0 0 1947 0 0 14 84 14 active+undersized+degraded 2023-11-02T08:42:52.920740+0000 34'98 66:144 [0,2] 0 [0,2] 0 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:01.841 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:44:01.841 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:44:01.841 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:44:01.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2023-11-02T08:44:01.845 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:44:01.845 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:44:01.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:44:01.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:44:01.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:44:01.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:44:01.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: expfile=td/divergent-priors/exp.85082.out 2023-11-02T08:44:01.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:798: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.85082.out 2023-11-02T08:44:01.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2023-11-02T08:44:01.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2023-11-02T08:44:01.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2023-11-02T08:44:01.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2023-11-02T08:44:01.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2023-11-02T08:44:01.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.85082.out 2023-11-02T08:44:02.861 INFO:tasks.workunit.client.0.smithi116.stderr:Exporting 1.1b info 1.1b( v 34'98 lc 0'0 (32'84,34'98] local-lis/les=71/72 n=4 ec=33/23 lis/c=71/39 les/c/f=72/40/0 sis=71) 2023-11-02T08:44:02.862 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:d9cbd1d6:::existing_64:head# 2023-11-02T08:44:02.872 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:dcc4ff82:::existing_98:head# 2023-11-02T08:44:02.873 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:dd677bd3:::existing_35:head# 2023-11-02T08:44:02.874 INFO:tasks.workunit.client.0.smithi116.stderr:Read #1:dd6d90e7:::existing_46:head# 2023-11-02T08:44:02.875 INFO:tasks.workunit.client.0.smithi116.stderr:Export successful 2023-11-02T08:44:02.875 INFO:tasks.workunit.client.0.smithi116.stdout: marking collection for removal 2023-11-02T08:44:02.875 INFO:tasks.workunit.client.0.smithi116.stdout:setting '_remove' omap key 2023-11-02T08:44:02.876 INFO:tasks.workunit.client.0.smithi116.stdout:finish_remove_pgs 1.1b_head removing 1.1b 2023-11-02T08:44:03.011 INFO:tasks.workunit.client.0.smithi116.stderr:Remove successful 2023-11-02T08:44:03.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.85082.out 2023-11-02T08:44:03.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2023-11-02T08:44:03.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2023-11-02T08:44:03.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2023-11-02T08:44:03.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2023-11-02T08:44:03.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2023-11-02T08:44:03.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.85082.out 2023-11-02T08:44:04.183 INFO:tasks.workunit.client.0.smithi116.stdout:get_pg_num_history pg_num_history pg_num_history(e73 pg_nums {1={23=1,33=32}} deleted_pools ) 2023-11-02T08:44:04.183 INFO:tasks.workunit.client.0.smithi116.stdout:Importing pgid 1.1b 2023-11-02T08:44:04.184 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:d9cbd1d6:::existing_64:head# 2023-11-02T08:44:04.184 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:44:04.239 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:dcc4ff82:::existing_98:head# 2023-11-02T08:44:04.240 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:44:04.264 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:dd677bd3:::existing_35:head# 2023-11-02T08:44:04.265 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:44:04.289 INFO:tasks.workunit.client.0.smithi116.stdout:Write #1:dd6d90e7:::existing_46:head# 2023-11-02T08:44:04.290 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 0=[]:{} 2023-11-02T08:44:04.315 INFO:tasks.workunit.client.0.smithi116.stdout:write_pg epoch 72 info 1.1b( v 34'98 lc 0'0 (32'84,34'98] local-lis/les=71/72 n=4 ec=33/23 lis/c=71/39 les/c/f=72/40/0 sis=71) 2023-11-02T08:44:04.340 INFO:tasks.workunit.client.0.smithi116.stdout:Import successful 2023-11-02T08:44:04.859 INFO:tasks.workunit.client.0.smithi116.stdout:reviving divergent 1 2023-11-02T08:44:04.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:801: TEST_divergent_3: echo 'reviving divergent 1' 2023-11-02T08:44:04.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: ceph pg dump pgs 2023-11-02T08:44:05.228 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:44:05.229 INFO:tasks.workunit.client.0.smithi116.stdout:1.1f 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:43:59.869084+0000 32'97 72:149 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.229 INFO:tasks.workunit.client.0.smithi116.stdout:1.1e 4 0 0 0 0 1947 0 0 14 84 14 active+clean 2023-11-02T08:43:59.868468+0000 34'98 72:145 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.229 INFO:tasks.workunit.client.0.smithi116.stdout:1.1d 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:00.276865+0000 32'97 73:148 [2,1,0] 2 [2,1,0] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.229 INFO:tasks.workunit.client.0.smithi116.stdout:1.b 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:43:59.869672+0000 32'97 72:149 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.230 INFO:tasks.workunit.client.0.smithi116.stdout:1.a 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:43:59.868750+0000 32'97 73:140 [1,2,0] 1 [1,2,0] 1 32'97 2023-11-02T08:42:20.386938+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.230 INFO:tasks.workunit.client.0.smithi116.stdout:1.9 3 1 1 0 0 1464 0 0 13 84 13 active+recovery_wait+degraded 2023-11-02T08:43:59.869689+0000 32'97 73:137 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.230 INFO:tasks.workunit.client.0.smithi116.stdout:1.8 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:43:59.867545+0000 32'97 72:157 [1,2,0] 1 [1,2,0] 1 32'97 2023-11-02T08:42:17.109289+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.230 INFO:tasks.workunit.client.0.smithi116.stdout:1.7 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:43:59.866813+0000 32'97 72:149 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.231 INFO:tasks.workunit.client.0.smithi116.stdout:1.6 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:43:59.868907+0000 32'97 72:143 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:19.366599+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.231 INFO:tasks.workunit.client.0.smithi116.stdout:1.5 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:00.276995+0000 32'97 73:148 [2,0,1] 2 [2,0,1] 2 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.231 INFO:tasks.workunit.client.0.smithi116.stdout:1.4 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:43:59.869386+0000 32'97 72:157 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:18.171099+0000 32'97 2023-11-02T08:42:18.171099+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.231 INFO:tasks.workunit.client.0.smithi116.stdout:1.3 4 0 0 0 0 1948 0 0 14 84 14 active+clean 2023-11-02T08:43:59.867358+0000 34'98 72:150 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.232 INFO:tasks.workunit.client.0.smithi116.stdout:1.2 1 0 0 0 0 483 0 0 13 84 13 active+clean 2023-11-02T08:44:00.276808+0000 32'97 73:166 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:42:12.415700+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T14:54:52.696258+0000 1 0 2023-11-02T08:44:05.232 INFO:tasks.workunit.client.0.smithi116.stdout:1.c 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:43:59.866614+0000 32'97 72:159 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:22.122799+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.232 INFO:tasks.workunit.client.0.smithi116.stdout:1.1 2 0 0 0 0 966 0 0 13 84 13 active+clean 2023-11-02T08:44:00.277909+0000 32'97 73:159 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:42:09.981464+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T17:11:35.158983+0000 2 0 2023-11-02T08:44:05.232 INFO:tasks.workunit.client.0.smithi116.stdout:1.1c 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:43:59.868048+0000 32'97 72:138 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.233 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:43:59.868016+0000 32'97 72:151 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:07.198292+0000 32'97 2023-11-02T08:42:07.198292+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.233 INFO:tasks.workunit.client.0.smithi116.stdout:1.d 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:00.276693+0000 32'97 73:148 [2,1,0] 2 [2,1,0] 2 32'97 2023-11-02T08:42:19.844682+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.233 INFO:tasks.workunit.client.0.smithi116.stdout:1.e 3 1 1 0 0 1111 0 0 14 84 14 active+recovery_wait+degraded 2023-11-02T08:43:59.869014+0000 66'98 72:137 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.233 INFO:tasks.workunit.client.0.smithi116.stdout:1.f 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:43:59.868180+0000 32'97 72:149 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.234 INFO:tasks.workunit.client.0.smithi116.stdout:1.10 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:43:59.869275+0000 32'97 72:160 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:23.146709+0000 32'97 2023-11-02T08:42:23.146709+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.234 INFO:tasks.workunit.client.0.smithi116.stdout:1.11 3 0 1 0 0 1464 0 0 13 84 13 active+recovery_wait+degraded 2023-11-02T08:44:00.276636+0000 32'97 73:147 [2,0,1] 2 [2,0,1] 2 39'98 2023-11-02T08:42:18.867158+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.234 INFO:tasks.workunit.client.0.smithi116.stdout:1.12 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:00.276703+0000 32'97 73:154 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:42:15.453129+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.234 INFO:tasks.workunit.client.0.smithi116.stdout:1.13 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:43:59.869573+0000 32'97 72:149 [1,2,0] 1 [1,2,0] 1 32'97 2023-11-02T08:42:10.923269+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.235 INFO:tasks.workunit.client.0.smithi116.stdout:1.14 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:43:59.867437+0000 32'97 72:138 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.235 INFO:tasks.workunit.client.0.smithi116.stdout:1.15 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:00.277731+0000 32'97 73:148 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:42:14.857410+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.235 INFO:tasks.workunit.client.0.smithi116.stdout:1.16 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:43:59.782443+0000 32'97 72:144 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:16.428517+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.235 INFO:tasks.workunit.client.0.smithi116.stdout:1.17 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:43:59.867872+0000 32'97 72:149 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.236 INFO:tasks.workunit.client.0.smithi116.stdout:1.18 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:43:59.869213+0000 32'97 72:138 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.236 INFO:tasks.workunit.client.0.smithi116.stdout:1.19 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:43:59.869036+0000 32'97 73:138 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.236 INFO:tasks.workunit.client.0.smithi116.stdout:1.1a 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:43:59.869481+0000 32'97 73:140 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.236 INFO:tasks.workunit.client.0.smithi116.stdout:1.1b 4 1 1 0 0 1947 0 0 14 84 14 active+recovering+degraded 2023-11-02T08:43:59.870134+0000 34'98 72:149 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:05.236 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:44:05.236 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:44:05.237 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:44:05.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: activate_osd td/divergent-priors 1 2023-11-02T08:44:05.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2023-11-02T08:44:05.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:44:05.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2023-11-02T08:44:05.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:44:05.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2023-11-02T08:44:05.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:44:05.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:44:05.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:44:05.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:44:05.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2023-11-02T08:44:05.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2023-11-02T08:44:05.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:44:05.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:44:05.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:44:05.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:44:05.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:44:05.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:44:05.245 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:44:05.245 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:44:05.246 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:44:05.246 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:44:05.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:44:05.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:44:05.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:44:05.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:44:05.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:44:05.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:44:05.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:44:05.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:44:05.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:44:05.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:44:05.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2023-11-02T08:44:05.248 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T08:44:05.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2023-11-02T08:44:05.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:44:05.250 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2023-11-02T08:44:05.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2023-11-02T08:44:05.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:44:05.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:44:05.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:44:05.276 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:44:05.272+0000 7f14af60e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:44:05.277 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:44:05.274+0000 7f14af60e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:44:05.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:44:05.276+0000 7f14af60e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:44:05.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2023-11-02T08:44:05.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:44:05.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:44:05.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:44:05.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:44:05.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:44:05.648 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:44:05.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:44:05.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:44:05.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:44:05.798 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:44:05.795+0000 7f14af60e500 -1 Falling back to public interface 2023-11-02T08:44:06.047 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 71 up_thru 71 down_at 46 last_clean_interval [10,44) [v2:127.0.0.1:6818/123641,v1:127.0.0.1:6819/123641] [v2:127.0.0.1:6820/123641,v1:127.0.0.1:6821/123641] exists,up 4d681ad5-42d6-443f-ac67-86739e3ca916 2023-11-02T08:44:06.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:44:06.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:44:06.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:44:06.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: wait_for_osd up 1 2023-11-02T08:44:06.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:44:06.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:44:06.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:44:06.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:44:06.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:44:06.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:44:06.051 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:44:06.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:44:06.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:44:06.425 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 71 up_thru 71 down_at 46 last_clean_interval [10,44) [v2:127.0.0.1:6818/123641,v1:127.0.0.1:6819/123641] [v2:127.0.0.1:6820/123641,v1:127.0.0.1:6821/123641] exists,up 4d681ad5-42d6-443f-ac67-86739e3ca916 2023-11-02T08:44:06.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:44:06.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:44:06.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:44:06.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: sleep 20 2023-11-02T08:44:08.345 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:44:08.343+0000 7f14af60e500 -1 osd.1 73 log_to_monitors true 2023-11-02T08:44:26.429 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: get_asok_path osd.1 2023-11-02T08:44:26.429 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-11-02T08:44:26.429 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-11-02T08:44:26.430 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:44:26.430 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:44:26.430 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:44:26.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.1.asok 2023-11-02T08:44:26.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: CEPH_ARGS= 2023-11-02T08:44:26.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.1.asok dump_ops_in_flight 2023-11-02T08:44:26.515 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T08:44:26.515 INFO:tasks.workunit.client.0.smithi116.stdout: "ops": [], 2023-11-02T08:44:26.516 INFO:tasks.workunit.client.0.smithi116.stdout: "num_ops": 0 2023-11-02T08:44:26.516 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T08:44:26.523 INFO:tasks.workunit.client.0.smithi116.stdout:allowing recovery 2023-11-02T08:44:26.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:809: TEST_divergent_3: echo 'allowing recovery' 2023-11-02T08:44:26.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: ceph pg dump pgs 2023-11-02T08:44:26.882 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:44:26.882 INFO:tasks.workunit.client.0.smithi116.stdout:1.1f 3 0 0 0 0 1449 0 0 13 84 13 active+clean 2023-11-02T08:44:21.055975+0000 32'97 86:161 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:44:21.055864+0000 32'97 2023-11-02T08:44:21.055864+0000 0 1 periodic scrub scheduled @ 2023-11-03T12:20:43.728723+0000 3 0 2023-11-02T08:44:26.882 INFO:tasks.workunit.client.0.smithi116.stdout:1.1e 4 0 0 0 0 1947 0 0 14 84 14 active+clean 2023-11-02T08:44:15.701829+0000 34'98 85:173 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:26.883 INFO:tasks.workunit.client.0.smithi116.stdout:1.1d 4 0 0 0 0 1932 0 0 13 84 13 active+clean 2023-11-02T08:44:16.995826+0000 32'97 86:183 [2,1,0] 2 [2,1,0] 2 32'97 2023-11-02T08:44:16.995729+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T13:06:32.244827+0000 4 0 2023-11-02T08:44:26.883 INFO:tasks.workunit.client.0.smithi116.stdout:1.b 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.821378+0000 32'97 85:168 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:26.883 INFO:tasks.workunit.client.0.smithi116.stdout:1.a 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.820664+0000 32'97 85:158 [1,2,0] 1 [1,2,0] 1 32'97 2023-11-02T08:42:20.386938+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:26.883 INFO:tasks.workunit.client.0.smithi116.stdout:1.9 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.990208+0000 32'97 85:160 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:26.883 INFO:tasks.workunit.client.0.smithi116.stdout:1.8 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.817694+0000 32'97 85:176 [1,2,0] 1 [1,2,0] 1 32'97 2023-11-02T08:42:17.109289+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:26.884 INFO:tasks.workunit.client.0.smithi116.stdout:1.7 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.815538+0000 32'97 85:168 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:26.884 INFO:tasks.workunit.client.0.smithi116.stdout:1.6 4 0 0 0 0 1947 0 0 14 84 14 active+clean 2023-11-02T08:44:09.822508+0000 73'98 85:163 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:19.366599+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:26.884 INFO:tasks.workunit.client.0.smithi116.stdout:1.5 6 0 0 0 0 2898 0 0 13 84 13 active+clean 2023-11-02T08:44:11.025801+0000 32'97 86:183 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:44:11.025710+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T15:40:40.690076+0000 6 0 2023-11-02T08:44:26.884 INFO:tasks.workunit.client.0.smithi116.stdout:1.4 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.821328+0000 32'97 85:176 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:18.171099+0000 32'97 2023-11-02T08:42:18.171099+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:26.884 INFO:tasks.workunit.client.0.smithi116.stdout:1.3 4 0 0 0 0 1948 0 0 14 84 14 active+clean 2023-11-02T08:44:09.817907+0000 34'98 85:169 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:26.885 INFO:tasks.workunit.client.0.smithi116.stdout:1.2 1 0 0 0 0 483 0 0 13 84 13 active+clean 2023-11-02T08:44:09.815643+0000 32'97 87:192 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:42:12.415700+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T15:49:23.141888+0000 1 0 2023-11-02T08:44:26.885 INFO:tasks.workunit.client.0.smithi116.stdout:1.c 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.815755+0000 32'97 85:178 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:22.122799+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:26.885 INFO:tasks.workunit.client.0.smithi116.stdout:1.1 2 0 0 0 0 966 0 0 13 84 13 active+clean 2023-11-02T08:44:09.816547+0000 32'97 86:182 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:42:09.981464+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T20:42:00.183449+0000 2 0 2023-11-02T08:44:26.885 INFO:tasks.workunit.client.0.smithi116.stdout:1.1c 2 0 0 0 0 966 0 0 13 84 13 active+clean 2023-11-02T08:44:14.427386+0000 32'97 85:169 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:44:14.427276+0000 32'97 2023-11-02T08:44:14.427276+0000 0 1 periodic scrub scheduled @ 2023-11-03T08:46:16.524280+0000 2 0 2023-11-02T08:44:26.885 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.821730+0000 32'97 85:170 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:07.198292+0000 32'97 2023-11-02T08:42:07.198292+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:26.886 INFO:tasks.workunit.client.0.smithi116.stdout:1.d 3 0 0 0 0 1449 0 0 13 84 13 active+clean 2023-11-02T08:44:18.019381+0000 32'97 86:183 [2,1,0] 2 [2,1,0] 2 32'97 2023-11-02T08:44:18.019292+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T11:44:31.300758+0000 3 0 2023-11-02T08:44:26.886 INFO:tasks.workunit.client.0.smithi116.stdout:1.e 3 0 0 0 0 1111 0 0 14 84 14 active+clean 2023-11-02T08:44:15.755744+0000 66'98 85:170 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:26.886 INFO:tasks.workunit.client.0.smithi116.stdout:1.f 4 0 0 0 0 1932 0 0 13 84 13 active+clean 2023-11-02T08:44:22.924760+0000 32'97 86:161 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:44:22.924633+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T17:02:13.342048+0000 4 0 2023-11-02T08:44:26.886 INFO:tasks.workunit.client.0.smithi116.stdout:1.10 3 0 0 0 0 1449 0 0 13 84 13 active+clean 2023-11-02T08:44:20.236752+0000 32'97 87:161 [0,2,1] 0 [0,2,1] 0 32'97 2023-11-02T08:44:20.236651+0000 32'97 2023-11-02T08:42:23.146709+0000 0 1 periodic scrub scheduled @ 2023-11-03T17:55:06.837142+0000 3 0 2023-11-02T08:44:26.886 INFO:tasks.workunit.client.0.smithi116.stdout:1.11 3 0 0 0 0 1449 0 0 13 84 13 active+clean+inconsistent 2023-11-02T08:44:22.292522+0000 32'97 88:148 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:44:22.292411+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T12:40:24.749375+0000 3 0 2023-11-02T08:44:26.887 INFO:tasks.workunit.client.0.smithi116.stdout:1.12 1 0 0 0 0 483 0 0 13 84 13 active+clean 2023-11-02T08:44:10.270169+0000 32'97 87:192 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:44:10.270076+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T15:12:09.947759+0000 1 0 2023-11-02T08:44:26.887 INFO:tasks.workunit.client.0.smithi116.stdout:1.13 3 0 0 0 0 1449 0 0 13 84 13 active+clean 2023-11-02T08:44:17.507576+0000 32'97 85:180 [1,2,0] 1 [1,2,0] 1 32'97 2023-11-02T08:44:17.507462+0000 32'97 2023-11-02T08:44:17.507462+0000 0 1 periodic scrub scheduled @ 2023-11-03T14:36:33.371486+0000 3 0 2023-11-02T08:44:26.887 INFO:tasks.workunit.client.0.smithi116.stdout:1.14 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.821974+0000 32'97 85:157 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:26.887 INFO:tasks.workunit.client.0.smithi116.stdout:1.15 3 0 0 0 0 1449 0 0 13 84 13 active+clean 2023-11-02T08:44:18.964510+0000 32'97 86:183 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:44:18.964408+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T11:06:53.382004+0000 3 0 2023-11-02T08:44:26.887 INFO:tasks.workunit.client.0.smithi116.stdout:1.16 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.817004+0000 32'97 85:163 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:16.428517+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:26.888 INFO:tasks.workunit.client.0.smithi116.stdout:1.17 1 0 0 0 0 483 0 0 13 84 13 active+clean 2023-11-02T08:44:20.367702+0000 32'97 85:180 [1,2,0] 1 [1,2,0] 1 32'97 2023-11-02T08:44:20.367609+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T10:15:24.822764+0000 1 0 2023-11-02T08:44:26.888 INFO:tasks.workunit.client.0.smithi116.stdout:1.18 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.821631+0000 32'97 85:157 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:26.888 INFO:tasks.workunit.client.0.smithi116.stdout:1.19 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.820448+0000 32'97 85:156 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:26.888 INFO:tasks.workunit.client.0.smithi116.stdout:1.1a 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.821637+0000 32'97 85:158 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:26.888 INFO:tasks.workunit.client.0.smithi116.stdout:1.1b 4 0 0 0 0 1947 0 0 14 84 14 active+clean 2023-11-02T08:44:10.090822+0000 34'98 85:172 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:26.889 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:44:26.889 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:44:26.889 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:44:26.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2023-11-02T08:44:26.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2023-11-02T08:44:27.080 INFO:tasks.workunit.client.0.smithi116.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2023-11-02T08:44:27.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2023-11-02T08:44:27.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2023-11-02T08:44:27.199 INFO:tasks.workunit.client.0.smithi116.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2023-11-02T08:44:27.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2023-11-02T08:44:27.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2023-11-02T08:44:27.319 INFO:tasks.workunit.client.0.smithi116.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2023-11-02T08:44:27.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:817: TEST_divergent_3: echo 'reading divergent objects' 2023-11-02T08:44:27.328 INFO:tasks.workunit.client.0.smithi116.stdout:reading divergent objects 2023-11-02T08:44:27.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:818: TEST_divergent_3: ceph pg dump pgs 2023-11-02T08:44:27.830 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:44:27.831 INFO:tasks.workunit.client.0.smithi116.stdout:1.1f 3 0 0 0 0 1449 0 0 13 84 13 active+clean 2023-11-02T08:44:21.055975+0000 32'97 86:161 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:44:21.055864+0000 32'97 2023-11-02T08:44:21.055864+0000 0 1 periodic scrub scheduled @ 2023-11-03T12:20:43.728723+0000 3 0 2023-11-02T08:44:27.831 INFO:tasks.workunit.client.0.smithi116.stdout:1.1e 4 0 0 0 0 1947 0 0 14 84 14 active+clean 2023-11-02T08:44:15.701829+0000 34'98 85:173 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:27.832 INFO:tasks.workunit.client.0.smithi116.stdout:1.1d 4 0 0 0 0 1932 0 0 13 84 13 active+clean 2023-11-02T08:44:16.995826+0000 32'97 86:183 [2,1,0] 2 [2,1,0] 2 32'97 2023-11-02T08:44:16.995729+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T13:06:32.244827+0000 4 0 2023-11-02T08:44:27.832 INFO:tasks.workunit.client.0.smithi116.stdout:1.b 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.821378+0000 32'97 85:168 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:27.832 INFO:tasks.workunit.client.0.smithi116.stdout:1.a 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.820664+0000 32'97 85:158 [1,2,0] 1 [1,2,0] 1 32'97 2023-11-02T08:42:20.386938+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:27.832 INFO:tasks.workunit.client.0.smithi116.stdout:1.9 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.990208+0000 32'97 85:160 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:27.833 INFO:tasks.workunit.client.0.smithi116.stdout:1.8 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.817694+0000 32'97 85:176 [1,2,0] 1 [1,2,0] 1 32'97 2023-11-02T08:42:17.109289+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:27.833 INFO:tasks.workunit.client.0.smithi116.stdout:1.7 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.815538+0000 32'97 85:168 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:27.833 INFO:tasks.workunit.client.0.smithi116.stdout:1.6 4 0 0 0 0 1947 0 0 14 84 14 active+clean 2023-11-02T08:44:09.822508+0000 73'98 85:163 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:19.366599+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:27.834 INFO:tasks.workunit.client.0.smithi116.stdout:1.5 6 0 0 0 0 2898 0 0 13 84 13 active+clean 2023-11-02T08:44:11.025801+0000 32'97 86:183 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:44:11.025710+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T15:40:40.690076+0000 6 0 2023-11-02T08:44:27.834 INFO:tasks.workunit.client.0.smithi116.stdout:1.4 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.821328+0000 32'97 85:176 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:18.171099+0000 32'97 2023-11-02T08:42:18.171099+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:27.834 INFO:tasks.workunit.client.0.smithi116.stdout:1.3 4 0 0 0 0 1948 0 0 14 84 14 active+clean 2023-11-02T08:44:09.817907+0000 34'98 85:169 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:27.834 INFO:tasks.workunit.client.0.smithi116.stdout:1.2 1 0 0 0 0 483 0 0 13 84 13 active+clean 2023-11-02T08:44:09.815643+0000 32'97 87:192 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:42:12.415700+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T15:49:23.141888+0000 1 0 2023-11-02T08:44:27.835 INFO:tasks.workunit.client.0.smithi116.stdout:1.c 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.815755+0000 32'97 85:178 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:22.122799+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:27.835 INFO:tasks.workunit.client.0.smithi116.stdout:1.1 2 0 0 0 0 966 0 0 13 84 13 active+clean 2023-11-02T08:44:09.816547+0000 32'97 86:182 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:42:09.981464+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T20:42:00.183449+0000 2 0 2023-11-02T08:44:27.835 INFO:tasks.workunit.client.0.smithi116.stdout:1.1c 2 0 0 0 0 966 0 0 13 84 13 active+clean 2023-11-02T08:44:14.427386+0000 32'97 85:169 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:44:14.427276+0000 32'97 2023-11-02T08:44:14.427276+0000 0 1 periodic scrub scheduled @ 2023-11-03T08:46:16.524280+0000 2 0 2023-11-02T08:44:27.836 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.821730+0000 32'97 85:170 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:07.198292+0000 32'97 2023-11-02T08:42:07.198292+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:27.837 INFO:tasks.workunit.client.0.smithi116.stdout:1.d 3 0 0 0 0 1449 0 0 13 84 13 active+clean 2023-11-02T08:44:18.019381+0000 32'97 86:183 [2,1,0] 2 [2,1,0] 2 32'97 2023-11-02T08:44:18.019292+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T11:44:31.300758+0000 3 0 2023-11-02T08:44:27.837 INFO:tasks.workunit.client.0.smithi116.stdout:1.e 3 0 0 0 0 1111 0 0 14 84 14 active+clean 2023-11-02T08:44:15.755744+0000 66'98 85:170 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:27.837 INFO:tasks.workunit.client.0.smithi116.stdout:1.f 4 0 0 0 0 1932 0 0 13 84 13 active+clean 2023-11-02T08:44:22.924760+0000 32'97 86:161 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:44:22.924633+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T17:02:13.342048+0000 4 0 2023-11-02T08:44:27.837 INFO:tasks.workunit.client.0.smithi116.stdout:1.10 3 0 0 0 0 1449 0 0 13 84 13 active+clean 2023-11-02T08:44:20.236752+0000 32'97 87:161 [0,2,1] 0 [0,2,1] 0 32'97 2023-11-02T08:44:20.236651+0000 32'97 2023-11-02T08:42:23.146709+0000 0 1 periodic scrub scheduled @ 2023-11-03T17:55:06.837142+0000 3 0 2023-11-02T08:44:27.838 INFO:tasks.workunit.client.0.smithi116.stdout:1.11 3 0 0 0 0 1449 0 0 13 84 13 active+clean+inconsistent 2023-11-02T08:44:22.292522+0000 32'97 88:148 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:44:22.292411+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T12:40:24.749375+0000 3 0 2023-11-02T08:44:27.838 INFO:tasks.workunit.client.0.smithi116.stdout:1.12 1 0 0 0 0 483 0 0 13 84 13 active+clean 2023-11-02T08:44:10.270169+0000 32'97 87:192 [0,1,2] 0 [0,1,2] 0 32'97 2023-11-02T08:44:10.270076+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T15:12:09.947759+0000 1 0 2023-11-02T08:44:27.838 INFO:tasks.workunit.client.0.smithi116.stdout:1.13 3 0 0 0 0 1449 0 0 13 84 13 active+clean 2023-11-02T08:44:17.507576+0000 32'97 85:180 [1,2,0] 1 [1,2,0] 1 32'97 2023-11-02T08:44:17.507462+0000 32'97 2023-11-02T08:44:17.507462+0000 0 1 periodic scrub scheduled @ 2023-11-03T14:36:33.371486+0000 3 0 2023-11-02T08:44:27.838 INFO:tasks.workunit.client.0.smithi116.stdout:1.14 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.821974+0000 32'97 85:157 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:27.839 INFO:tasks.workunit.client.0.smithi116.stdout:1.15 3 0 0 0 0 1449 0 0 13 84 13 active+clean 2023-11-02T08:44:18.964510+0000 32'97 86:183 [2,0,1] 2 [2,0,1] 2 32'97 2023-11-02T08:44:18.964408+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T11:06:53.382004+0000 3 0 2023-11-02T08:44:27.839 INFO:tasks.workunit.client.0.smithi116.stdout:1.16 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.817004+0000 32'97 85:163 [1,0,2] 1 [1,0,2] 1 32'97 2023-11-02T08:42:16.428517+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:27.839 INFO:tasks.workunit.client.0.smithi116.stdout:1.17 1 0 0 0 0 483 0 0 13 84 13 active+clean 2023-11-02T08:44:20.367702+0000 32'97 85:180 [1,2,0] 1 [1,2,0] 1 32'97 2023-11-02T08:44:20.367609+0000 0'0 2023-11-02T08:41:38.497171+0000 0 1 periodic scrub scheduled @ 2023-11-03T10:15:24.822764+0000 1 0 2023-11-02T08:44:27.839 INFO:tasks.workunit.client.0.smithi116.stdout:1.18 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.821631+0000 32'97 85:157 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:27.839 INFO:tasks.workunit.client.0.smithi116.stdout:1.19 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.820448+0000 32'97 85:156 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:27.840 INFO:tasks.workunit.client.0.smithi116.stdout:1.1a 3 0 0 0 0 1464 0 0 13 84 13 active+clean 2023-11-02T08:44:09.821637+0000 32'97 85:158 [1,2,0] 1 [1,2,0] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:27.840 INFO:tasks.workunit.client.0.smithi116.stdout:1.1b 4 0 0 0 0 1947 0 0 14 84 14 active+clean 2023-11-02T08:44:10.090822+0000 34'98 85:172 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:41:38.497171+0000 0'0 2023-11-02T08:41:38.497171+0000 0 0 queued for scrub 0 0 2023-11-02T08:44:27.840 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:44:27.840 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:44:27.841 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:44:27.842 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: expr 2 + 2 2023-11-02T08:44:27.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: seq 1 4 2023-11-02T08:44:27.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2023-11-02T08:44:27.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2023-11-02T08:44:27.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2023-11-02T08:44:27.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2023-11-02T08:44:27.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2023-11-02T08:44:27.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2023-11-02T08:44:27.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2023-11-02T08:44:27.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2023-11-02T08:44:27.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: seq 1 2 2023-11-02T08:44:27.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2023-11-02T08:44:27.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2023-11-02T08:44:28.021 INFO:tasks.workunit.client.0.smithi116.stderr:error getting test/newobject_1: (2) No such file or directory 2023-11-02T08:44:28.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2023-11-02T08:44:28.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2023-11-02T08:44:28.054 INFO:tasks.workunit.client.0.smithi116.stderr:error getting test/newobject_2: (2) No such file or directory 2023-11-02T08:44:28.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: rm -f td/divergent-priors/existing 2023-11-02T08:44:28.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2023-11-02T08:44:28.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log 2023-11-02T08:44:28.085 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.2.log:2023-11-02T08:43:59.341+0000 7f9636091640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 39'98 (31'3) delete 1:8881e6d8:::existing_3:head by client.4621.0:1 2023-11-02T08:42:13.145432+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2023-11-02T08:44:28.086 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.2.log:2023-11-02T08:43:59.341+0000 7f9636091640 20 _merge_object_divergent_entries: keeping 39'98 (31'3) delete 1:8881e6d8:::existing_3:head by client.4621.0:1 2023-11-02T08:42:13.145432+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2023-11-02T08:44:28.086 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.2.log:2023-11-02T08:43:59.341+0000 7f9636091640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2023-11-02T08:44:28.086 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.2.log:2023-11-02T08:43:59.341+0000 7f9636091640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 39'98 last_divergent_update: 39'98 2023-11-02T08:44:28.087 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.2.log:2023-11-02T08:43:59.341+0000 7f9636091640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2023-11-02T08:44:28.087 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.2.log:2023-11-02T08:43:59.341+0000 7f9636091640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2023-11-02T08:44:28.087 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.2.log:2023-11-02T08:43:59.341+0000 7f9636091640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 39'98 2023-11-02T08:44:28.087 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.2.log:2023-11-02T08:43:59.341+0000 7f9636091640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 39'98 (31'3) delete 1:8881e6d8:::existing_3:head by client.4621.0:1 2023-11-02T08:42:13.145432+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2023-11-02T08:44:28.088 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.2.log:2023-11-02T08:43:59.341+0000 7f9636091640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2023-11-02T08:44:28.088 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.2.log:2023-11-02T08:43:59.341+0000 7f9636091640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 32'84 2023-11-02T08:44:28.088 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74cb83a640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 40'99 (31'1) modify 1:d858c035:::existing_1:head by client.4626.0:1 2023-11-02T08:42:13.158389+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:44:28.088 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74cb83a640 20 _merge_object_divergent_entries: keeping 40'99 (31'1) modify 1:d858c035:::existing_1:head by client.4626.0:1 2023-11-02T08:42:13.158389+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:44:28.089 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74cb83a640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2023-11-02T08:44:28.089 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74cb83a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 31'1 first_divergent_update: 40'99 last_divergent_update: 40'99 2023-11-02T08:44:28.089 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74cb83a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2023-11-02T08:44:28.089 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74cb83a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2023-11-02T08:44:28.089 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74cb83a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 40'100 2023-11-02T08:44:28.090 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74cb83a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 40'99 (31'1) modify 1:d858c035:::existing_1:head by client.4626.0:1 2023-11-02T08:42:13.158389+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:44:28.090 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74cb83a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2023-11-02T08:44:28.090 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74cb83a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 31'1 <= info.log_tail 32'84 2023-11-02T08:44:28.090 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74cb83a640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 40'100 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4629.0:1 2023-11-02T08:42:13.161132+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2023-11-02T08:44:28.090 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74cb83a640 20 _merge_object_divergent_entries: keeping 40'100 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4629.0:1 2023-11-02T08:42:13.161132+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2023-11-02T08:44:28.091 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74cb83a640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2023-11-02T08:44:28.091 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74cb83a640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 40'100 last_divergent_update: 40'100 2023-11-02T08:44:28.091 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74cb83a640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2023-11-02T08:44:28.091 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74cb83a640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2023-11-02T08:44:28.092 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74ca838640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 39'99 (31'4) delete 1:70b3bf12:::existing_4:head by client.4631.0:1 2023-11-02T08:42:13.161640+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2023-11-02T08:44:28.092 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74ca838640 20 _merge_object_divergent_entries: keeping 39'99 (31'4) delete 1:70b3bf12:::existing_4:head by client.4631.0:1 2023-11-02T08:42:13.161640+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2023-11-02T08:44:28.092 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74ca838640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2023-11-02T08:44:28.093 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74ca838640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 31'4 first_divergent_update: 39'99 last_divergent_update: 39'99 2023-11-02T08:44:28.093 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74ca838640 10 _merge_object_divergent_entries: more recent entry found: 66'98 (31'4) modify 1:70b3bf12:::existing_4:head by client.4748.0:1 2023-11-02T08:43:14.464395+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2023-11-02T08:44:28.093 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74ca838640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 39'98 (0'0) modify 1:75ac5091:::newobject_2:head by client.4619.0:1 2023-11-02T08:42:13.141468+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2023-11-02T08:44:28.093 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74ca838640 20 _merge_object_divergent_entries: keeping 39'98 (0'0) modify 1:75ac5091:::newobject_2:head by client.4619.0:1 2023-11-02T08:42:13.141468+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2023-11-02T08:44:28.094 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74ca838640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2023-11-02T08:44:28.094 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74ca838640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 39'98 last_divergent_update: 39'98 2023-11-02T08:44:28.095 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74ca838640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2023-11-02T08:44:28.095 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.343+0000 7f74ca838640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2023-11-02T08:44:28.095 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.403+0000 7f74ca838640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 39'98 (31'2) modify 1:97db2a0f:::existing_2:head by client.4633.0:1 2023-11-02T08:42:13.259396+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:44:28.095 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.403+0000 7f74ca838640 20 _merge_object_divergent_entries: keeping 39'98 (31'2) modify 1:97db2a0f:::existing_2:head by client.4633.0:1 2023-11-02T08:42:13.259396+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:44:28.096 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.403+0000 7f74ca838640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2023-11-02T08:44:28.096 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.403+0000 7f74ca838640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 31'2 first_divergent_update: 39'98 last_divergent_update: 39'98 2023-11-02T08:44:28.096 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.403+0000 7f74ca838640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2023-11-02T08:44:28.096 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.403+0000 7f74ca838640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2023-11-02T08:44:28.096 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.403+0000 7f74ca838640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 39'98 2023-11-02T08:44:28.097 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.403+0000 7f74ca838640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 39'98 (31'2) modify 1:97db2a0f:::existing_2:head by client.4633.0:1 2023-11-02T08:42:13.259396+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:44:28.097 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.403+0000 7f74ca838640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2023-11-02T08:44:28.097 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:43:59.403+0000 7f74ca838640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 31'2 <= info.log_tail 32'84 2023-11-02T08:44:28.097 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:44:00.082+0000 7f74cb83a640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 39'98 (31'3) delete 1:8881e6d8:::existing_3:head by client.4621.0:1 2023-11-02T08:42:13.145432+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2023-11-02T08:44:28.098 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:44:00.082+0000 7f74cb83a640 20 _merge_object_divergent_entries: keeping 39'98 (31'3) delete 1:8881e6d8:::existing_3:head by client.4621.0:1 2023-11-02T08:42:13.145432+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2023-11-02T08:44:28.098 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:44:00.082+0000 7f74cb83a640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2023-11-02T08:44:28.098 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:44:00.082+0000 7f74cb83a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 39'98 last_divergent_update: 39'98 2023-11-02T08:44:28.098 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:44:00.082+0000 7f74cb83a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2023-11-02T08:44:28.098 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:44:00.082+0000 7f74cb83a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2023-11-02T08:44:28.099 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:44:00.082+0000 7f74cb83a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 39'98 2023-11-02T08:44:28.099 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:44:00.082+0000 7f74cb83a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 39'98 (31'3) delete 1:8881e6d8:::existing_3:head by client.4621.0:1 2023-11-02T08:42:13.145432+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2023-11-02T08:44:28.099 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:44:00.082+0000 7f74cb83a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2023-11-02T08:44:28.099 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:44:00.082+0000 7f74cb83a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 32'84 2023-11-02T08:44:28.100 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2023-11-02T08:44:28.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log 2023-11-02T08:44:28.118 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2023-11-02T08:44:28.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log 2023-11-02T08:44:28.145 INFO:tasks.workunit.client.0.smithi116.stdout:success 2023-11-02T08:44:28.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:842: TEST_divergent_3: echo success 2023-11-02T08:44:28.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:844: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.85082.out 2023-11-02T08:44:28.146 INFO:tasks.workunit.client.0.smithi116.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.85082.out': No such file or directory 2023-11-02T08:44:28.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:846: TEST_divergent_3: delete_pool test 2023-11-02T08:44:28.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2023-11-02T08:44:28.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2023-11-02T08:44:28.705 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test' does not exist 2023-11-02T08:44:28.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:847: TEST_divergent_3: kill_daemons td/divergent-priors 2023-11-02T08:44:28.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:44:28.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:44:28.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:44:28.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:44:28.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:44:34.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:44:34.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2023-11-02T08:44:34.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2023-11-02T08:44:34.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T08:44:34.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2023-11-02T08:44:34.032 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:44:34.032 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:44:34.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:44:34.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:44:34.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:44:34.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:44:34.038 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T08:44:34.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T08:44:34.039 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T08:44:34.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T08:44:34.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T08:44:34.042 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T08:44:34.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:44:34.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T08:44:34.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T08:44:34.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:44:34.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T08:44:34.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T08:44:34.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2023-11-02T08:44:34.077 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T08:44:34.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:44:34.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:44:34.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.85082 2023-11-02T08:44:34.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T08:44:34.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T08:44:34.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2023-11-02T08:44:34.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2023-11-02T08:44:34.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2023-11-02T08:44:34.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2023-11-02T08:44:34.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2023-11-02T08:44:34.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T08:44:34.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2023-11-02T08:44:34.082 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:44:34.082 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:44:34.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:44:34.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:44:34.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:44:34.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:44:34.084 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T08:44:34.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T08:44:34.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T08:44:34.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T08:44:34.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T08:44:34.086 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T08:44:34.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:44:34.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T08:44:34.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T08:44:34.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:44:34.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T08:44:34.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T08:44:34.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2023-11-02T08:44:34.089 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T08:44:34.089 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:44:34.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:44:34.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.85082 2023-11-02T08:44:34.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T08:44:34.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T08:44:34.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2023-11-02T08:44:34.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T08:44:34.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:44:34.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:44:34.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.85082 2023-11-02T08:44:34.095 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T08:44:34.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-11-02T08:44:34.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T08:44:34.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2023-11-02T08:44:34.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2023-11-02T08:44:34.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2023-11-02T08:44:34.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2023-11-02T08:44:34.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2023-11-02T08:44:34.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2023-11-02T08:44:34.096 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2023-11-02T08:44:34.097 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2023-11-02T08:44:34.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2023-11-02T08:44:34.098 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:44:34.098 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:44:34.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2023-11-02T08:44:34.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2023-11-02T08:44:34.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T08:44:34.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T08:44:34.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T08:44:34.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2023-11-02T08:44:34.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2023-11-02T08:44:34.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T08:44:34.473 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:44:34.473 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:44:34.473 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:44:34.473 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:44:34.473 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:44:34.474 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:44:34.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T08:44:34.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T08:44:34.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T08:44:34.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T08:44:34.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T08:44:34.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T08:44:34.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T08:44:34.666 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T08:44:34.667 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T08:44:34.667 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T08:44:34.668 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:44:34.668 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:44:34.668 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:44:34.668 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-mon.a.asok 2023-11-02T08:44:34.669 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T08:44:34.669 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.85082/ceph-mon.a.asok config get fsid 2023-11-02T08:44:34.747 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T08:44:34.747 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T08:44:34.747 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T08:44:34.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T08:44:34.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T08:44:34.748 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T08:44:34.748 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T08:44:34.748 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T08:44:34.749 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:44:34.749 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:44:34.749 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:44:34.749 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-mon.a.asok 2023-11-02T08:44:34.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T08:44:34.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.85082/ceph-mon.a.asok config get mon_host 2023-11-02T08:44:34.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2023-11-02T08:44:34.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2023-11-02T08:44:34.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T08:44:34.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T08:44:34.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T08:44:34.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2023-11-02T08:44:34.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T08:44:35.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T08:44:35.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:44:35.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:44:35.678 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:44:35.678 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:44:35.678 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:44:35.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:44:35.679 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T08:44:35.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T08:44:35.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2023-11-02T08:44:35.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2023-11-02T08:44:35.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2023-11-02T08:44:35.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:44:35.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T08:44:35.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:44:35.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2023-11-02T08:44:35.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:44:35.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:44:35.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:44:35.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:44:35.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2023-11-02T08:44:35.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2023-11-02T08:44:35.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:44:35.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:44:35.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:44:35.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:44:35.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:44:35.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:44:35.716 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:44:35.717 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:44:35.717 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:44:35.717 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:44:35.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:44:35.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:44:35.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:44:35.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:44:35.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:44:35.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:44:35.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:44:35.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:44:35.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:44:35.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:44:35.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:44:35.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:44:35.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2023-11-02T08:44:35.720 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:44:35.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5f38bac9-3e89-4df4-9a41-799f865c97c8 2023-11-02T08:44:35.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5f38bac9-3e89-4df4-9a41-799f865c97c8' 2023-11-02T08:44:35.721 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 5f38bac9-3e89-4df4-9a41-799f865c97c8 2023-11-02T08:44:35.721 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:44:35.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBzYUNl5YCgKxAAqH3MdwOPY4S3KQswZ+W6zA== 2023-11-02T08:44:35.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBzYUNl5YCgKxAAqH3MdwOPY4S3KQswZ+W6zA=="}' 2023-11-02T08:44:35.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5f38bac9-3e89-4df4-9a41-799f865c97c8 -i td/divergent-priors/0/new.json 2023-11-02T08:44:36.065 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:44:36.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2023-11-02T08:44:36.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBzYUNl5YCgKxAAqH3MdwOPY4S3KQswZ+W6zA== --osd-uuid 5f38bac9-3e89-4df4-9a41-799f865c97c8 2023-11-02T08:44:36.111 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:44:36.108+0000 7f9de4eb6500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:44:36.113 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:44:36.110+0000 7f9de4eb6500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:44:36.114 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:44:36.111+0000 7f9de4eb6500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:44:36.114 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:44:36.111+0000 7f9de4eb6500 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2023-11-02T08:44:36.115 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:44:36.111+0000 7f9de4eb6500 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2023-11-02T08:44:36.115 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:44:36.111+0000 7f9de4eb6500 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2023-11-02T08:44:36.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:44:36.194+0000 7f9de4eb6500 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2023-11-02T08:44:41.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2023-11-02T08:44:41.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:44:41.447 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T08:44:41.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T08:44:41.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:44:41.981 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T08:44:41.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T08:44:41.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:44:41.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:44:41.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:44:41.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:44:42.014 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:44:42.010+0000 7f1f2f40d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:44:42.037 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:44:42.033+0000 7f1f2f40d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:44:42.037 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:44:42.034+0000 7f1f2f40d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:44:42.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T08:44:42.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:44:42.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:44:42.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:44:42.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:44:42.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:44:42.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:44:42.444 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:44:42.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:44:42.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:44:42.571 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:44:42.567+0000 7f1f2f40d500 -1 Falling back to public interface 2023-11-02T08:44:42.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:44:43.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:44:43.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:44:43.852 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:44:43.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:44:43.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:44:43.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:44:44.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:44:44.888 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:44:44.885+0000 7f1f2f40d500 -1 osd.0 0 log_to_monitors true 2023-11-02T08:44:45.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:44:45.278 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:44:45.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:44:45.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:44:45.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:44:45.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:44:45.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:44:46.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:44:46.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:44:46.794 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:44:46.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:44:46.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:44:46.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:44:47.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:44:48.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:44:48.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:44:48.202 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:44:48.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:44:48.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:44:48.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:44:48.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:44:49.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:44:49.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:44:49.712 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:44:49.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:44:49.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:44:49.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:44:50.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:44:51.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:44:51.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:44:51.118 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:44:51.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:44:51.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:44:51.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:44:51.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:44:52.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:44:52.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:44:52.589 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:44:52.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:44:52.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:44:52.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:44:52.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:44:53.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:44:53.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:44:53.977 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:44:53.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:44:53.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:44:53.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:44:54.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:44:55.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:44:55.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:44:55.376 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:44:55.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:44:55.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:44:55.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:44:55.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:44:56.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:44:56.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:44:56.817 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:44:56.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:44:56.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:44:56.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:44:57.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:44:58.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:44:58.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:44:58.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:44:58.545 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:44:58.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:44:58.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:44:59.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:00.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:00.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:00.125 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:45:00.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:45:00.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:00.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:45:00.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:01.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:01.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:01.877 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:45:01.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:45:01.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:01.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:45:02.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:03.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:03.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:03.560 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:45:03.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:45:03.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:03.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:45:04.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:05.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:05.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:05.727 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:45:05.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:45:05.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:05.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:45:06.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:07.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:07.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:07.226 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:45:07.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:45:07.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:07.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:45:07.653 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/127038,v1:127.0.0.1:6803/127038] [v2:127.0.0.1:6804/127038,v1:127.0.0.1:6805/127038] exists,up 5f38bac9-3e89-4df4-9a41-799f865c97c8 2023-11-02T08:45:07.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:45:07.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:45:07.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:45:07.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2023-11-02T08:45:07.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2023-11-02T08:45:07.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2023-11-02T08:45:07.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:45:07.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-11-02T08:45:07.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:45:07.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2023-11-02T08:45:07.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:45:07.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:45:07.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:45:07.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:45:07.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2023-11-02T08:45:07.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2023-11-02T08:45:07.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:45:07.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:45:07.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:45:07.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:45:07.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:45:07.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:45:07.666 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:45:07.667 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:45:07.667 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:45:07.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:45:07.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:45:07.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:45:07.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:45:07.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:45:07.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:45:07.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:45:07.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:45:07.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:45:07.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:45:07.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:45:07.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:45:07.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:45:07.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2023-11-02T08:45:07.670 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:45:07.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=00de077e-ecc3-49b3-b8fc-83cfdd948349 2023-11-02T08:45:07.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 00de077e-ecc3-49b3-b8fc-83cfdd948349' 2023-11-02T08:45:07.671 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 00de077e-ecc3-49b3-b8fc-83cfdd948349 2023-11-02T08:45:07.671 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:45:07.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTYUNlD6ZXKBAAE3b9sbaJd8LBWsJ1SNYtMA== 2023-11-02T08:45:07.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTYUNlD6ZXKBAAE3b9sbaJd8LBWsJ1SNYtMA=="}' 2023-11-02T08:45:07.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 00de077e-ecc3-49b3-b8fc-83cfdd948349 -i td/divergent-priors/1/new.json 2023-11-02T08:45:08.178 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:45:08.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2023-11-02T08:45:08.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTYUNlD6ZXKBAAE3b9sbaJd8LBWsJ1SNYtMA== --osd-uuid 00de077e-ecc3-49b3-b8fc-83cfdd948349 2023-11-02T08:45:08.224 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:08.221+0000 7f53dea49500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:45:08.225 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:08.223+0000 7f53dea49500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:45:08.226 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:08.224+0000 7f53dea49500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:45:08.227 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:08.224+0000 7f53dea49500 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2023-11-02T08:45:08.227 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:08.224+0000 7f53dea49500 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2023-11-02T08:45:08.227 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:08.224+0000 7f53dea49500 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2023-11-02T08:45:08.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:08.299+0000 7f53dea49500 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2023-11-02T08:45:13.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2023-11-02T08:45:13.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:45:13.738 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2023-11-02T08:45:13.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-11-02T08:45:13.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:45:14.261 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T08:45:14.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-11-02T08:45:14.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:45:14.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:45:14.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:45:14.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:45:14.294 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:14.291+0000 7f78c5615500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:45:14.307 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:14.304+0000 7f78c5615500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:45:14.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:14.305+0000 7f78c5615500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:45:14.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-11-02T08:45:14.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:45:14.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:45:14.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:45:14.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:45:14.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:14.714 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:45:14.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:45:14.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:14.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:45:14.839 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:14.836+0000 7f78c5615500 -1 Falling back to public interface 2023-11-02T08:45:15.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:16.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:16.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:16.132 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:45:16.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:45:16.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:16.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:45:16.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:17.092 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:17.089+0000 7f78c5615500 -1 osd.1 0 log_to_monitors true 2023-11-02T08:45:17.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:17.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:17.559 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:45:17.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:45:17.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:17.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:45:17.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:18.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:18.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:18.996 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:45:18.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:45:18.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:18.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:45:19.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:20.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:20.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:20.386 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:45:20.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:45:20.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:20.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:45:20.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:21.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:21.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:21.778 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:45:21.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:45:21.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:21.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:45:22.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:23.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:23.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:23.169 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:45:23.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:45:23.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:23.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:45:23.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:24.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:24.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:24.570 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:45:24.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:45:24.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:24.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:45:25.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:26.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:26.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:26.026 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:45:26.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:45:26.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:26.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:45:26.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:27.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:27.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:27.417 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:45:27.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:45:27.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:27.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:45:27.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:28.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:28.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:28.830 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:45:28.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:45:28.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:28.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:45:29.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:30.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:30.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:30.367 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:45:30.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:45:30.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:30.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:45:31.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:32.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:32.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:32.239 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:45:32.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:45:32.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:32.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:45:33.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:34.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:34.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:34.027 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:45:34.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:45:34.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:34.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:45:35.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:36.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:36.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:36.035 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:45:36.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:45:36.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:36.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:45:36.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:37.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:37.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:37.502 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:45:37.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:45:37.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:37.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:45:38.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:39.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:39.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:39.064 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:45:39.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:45:39.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:39.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:45:39.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:40.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:40.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:40.608 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T08:45:40.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T08:45:40.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:40.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:45:40.987 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/128752,v1:127.0.0.1:6811/128752] [v2:127.0.0.1:6812/128752,v1:127.0.0.1:6813/128752] exists,up 00de077e-ecc3-49b3-b8fc-83cfdd948349 2023-11-02T08:45:40.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:45:40.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:45:40.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:45:40.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2023-11-02T08:45:40.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2023-11-02T08:45:40.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2023-11-02T08:45:40.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:45:40.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-11-02T08:45:40.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:45:40.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2023-11-02T08:45:40.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:45:40.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:45:40.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:45:40.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:45:40.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2023-11-02T08:45:40.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2023-11-02T08:45:40.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:45:40.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:45:40.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:45:40.996 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:45:40.996 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:45:40.996 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:45:40.996 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:45:40.996 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:45:40.997 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:45:40.997 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:45:40.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:45:40.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:45:40.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:45:40.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:45:40.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:45:40.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:45:40.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:45:41.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:45:41.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:45:41.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:45:41.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:45:41.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:45:41.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2023-11-02T08:45:41.001 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:45:41.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c3024fa8-7c9d-4674-b3d4-e8b9122765a2 2023-11-02T08:45:41.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c3024fa8-7c9d-4674-b3d4-e8b9122765a2' 2023-11-02T08:45:41.003 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 c3024fa8-7c9d-4674-b3d4-e8b9122765a2 2023-11-02T08:45:41.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:45:41.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC1YUNl6o6jABAAHuEN/H3NhIc5qlLFJLiuAQ== 2023-11-02T08:45:41.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC1YUNl6o6jABAAHuEN/H3NhIc5qlLFJLiuAQ=="}' 2023-11-02T08:45:41.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c3024fa8-7c9d-4674-b3d4-e8b9122765a2 -i td/divergent-priors/2/new.json 2023-11-02T08:45:41.529 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:45:41.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2023-11-02T08:45:41.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1YUNl6o6jABAAHuEN/H3NhIc5qlLFJLiuAQ== --osd-uuid c3024fa8-7c9d-4674-b3d4-e8b9122765a2 2023-11-02T08:45:41.575 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:41.572+0000 7f893e0c5500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:45:41.578 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:41.575+0000 7f893e0c5500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:45:41.579 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:41.576+0000 7f893e0c5500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:45:41.579 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:41.576+0000 7f893e0c5500 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2023-11-02T08:45:41.580 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:41.576+0000 7f893e0c5500 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2023-11-02T08:45:41.580 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:41.576+0000 7f893e0c5500 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2023-11-02T08:45:41.662 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:41.659+0000 7f893e0c5500 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2023-11-02T08:45:47.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2023-11-02T08:45:47.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:45:47.198 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2023-11-02T08:45:47.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-11-02T08:45:47.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:45:47.719 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T08:45:47.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-11-02T08:45:47.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:45:47.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:45:47.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:45:47.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:45:47.752 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:47.748+0000 7f67eaca5500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:45:47.767 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:47.763+0000 7f67eaca5500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:45:47.768 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:47.764+0000 7f67eaca5500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:45:48.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-11-02T08:45:48.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:45:48.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T08:45:48.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:45:48.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:45:48.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:48.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:45:48.173 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:45:48.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:48.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:45:48.297 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:48.293+0000 7f67eaca5500 -1 Falling back to public interface 2023-11-02T08:45:48.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:49.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:49.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:49.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:45:49.583 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:45:49.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:49.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:45:49.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:50.587 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:45:50.584+0000 7f67eaca5500 -1 osd.2 0 log_to_monitors true 2023-11-02T08:45:50.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:50.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:50.974 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:45:50.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:45:50.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:50.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:45:51.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:52.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:52.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:52.422 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:45:52.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:45:52.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:52.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:45:52.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:53.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:53.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:53.936 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:45:53.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:45:53.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:53.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:45:54.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:55.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:55.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:55.396 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:45:55.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:45:55.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:55.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:45:55.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:56.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:56.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:56.817 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:45:56.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:45:56.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:56.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:45:57.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:58.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:58.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:58.375 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:45:58.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:45:58.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:58.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:45:58.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:45:59.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:45:59.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:45:59.767 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:45:59.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:45:59.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:45:59.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:46:00.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:46:01.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:46:01.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:46:01.236 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:46:01.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:46:01.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:46:01.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:46:01.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:46:02.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:46:02.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:46:02.658 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:46:02.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:46:02.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:46:02.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:46:03.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:46:04.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:46:04.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:46:04.343 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:46:04.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:46:04.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:46:04.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:46:04.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:46:05.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:46:05.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:46:05.902 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:46:05.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:46:05.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:46:05.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:46:06.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:46:07.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:46:07.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:46:07.862 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:46:07.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:46:07.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:46:07.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:46:08.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:46:09.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:46:09.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:46:09.452 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:46:09.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:46:09.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:46:09.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:46:10.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:46:11.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:46:11.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:46:11.253 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:46:11.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:46:11.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:46:11.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:46:11.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:46:12.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:46:12.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:46:12.678 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:46:12.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:46:12.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:46:12.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:46:13.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:46:14.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:46:14.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:46:14.156 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T08:46:14.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T08:46:14.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:46:14.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:46:14.576 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/130490,v1:127.0.0.1:6819/130490] [v2:127.0.0.1:6820/130490,v1:127.0.0.1:6821/130490] exists,up c3024fa8-7c9d-4674-b3d4-e8b9122765a2 2023-11-02T08:46:14.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:46:14.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:46:14.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:46:14.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2023-11-02T08:46:15.368 INFO:tasks.workunit.client.0.smithi116.stderr:noout is set 2023-11-02T08:46:15.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2023-11-02T08:46:16.088 INFO:tasks.workunit.client.0.smithi116.stderr:noin is set 2023-11-02T08:46:16.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2023-11-02T08:46:16.902 INFO:tasks.workunit.client.0.smithi116.stderr:nodown is set 2023-11-02T08:46:16.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2023-11-02T08:46:16.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2390: create_ec_pool: local pool_name=test 2023-11-02T08:46:16.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2391: create_ec_pool: shift 2023-11-02T08:46:16.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2392: create_ec_pool: local allow_overwrites=true 2023-11-02T08:46:16.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: create_ec_pool: shift 2023-11-02T08:46:16.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2023-11-02T08:46:17.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: create_ec_pool: create_pool test 1 1 erasure myprofile 2023-11-02T08:46:17.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2023-11-02T08:46:18.304 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test' already exists 2023-11-02T08:46:18.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T08:46:19.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: create_ec_pool: '[' true = true ']' 2023-11-02T08:46:19.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2023-11-02T08:46:20.152 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 allow_ec_overwrites to true 2023-11-02T08:46:20.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: create_ec_pool: wait_for_clean 2023-11-02T08:46:20.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T08:46:20.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T08:46:20.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T08:46:20.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-11-02T08:46:20.165 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T08:46:20.165 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T08:46:20.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T08:46:20.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T08:46:20.166 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T08:46:20.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-11-02T08:46:20.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T08:46:20.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T08:46:20.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T08:46:20.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:46:20.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:46:20.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:46:20.665 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:46:20.665 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:46:20.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:46:20.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:46:20.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:46:20.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836503 2023-11-02T08:46:20.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836503 2023-11-02T08:46:20.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836503' 2023-11-02T08:46:20.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:46:20.785 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:46:20.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672972 2023-11-02T08:46:20.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672972 2023-11-02T08:46:20.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836503 1-42949672972' 2023-11-02T08:46:20.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:46:20.904 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:46:21.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509447 2023-11-02T08:46:21.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509447 2023-11-02T08:46:21.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836503 1-42949672972 2-64424509447' 2023-11-02T08:46:21.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:46:21.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836503 2023-11-02T08:46:21.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:46:21.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:46:21.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836503 2023-11-02T08:46:21.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:46:21.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836503 2023-11-02T08:46:21.025 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836503 2023-11-02T08:46:21.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2023-11-02T08:46:21.026 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:46:21.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836501 -lt 21474836503 2023-11-02T08:46:21.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T08:46:22.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T08:46:22.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:46:22.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836503 -lt 21474836503 2023-11-02T08:46:22.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:46:22.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672972 2023-11-02T08:46:22.854 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:46:22.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T08:46:22.855 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672972 2023-11-02T08:46:22.855 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:46:22.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672972 2023-11-02T08:46:22.856 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672972 2023-11-02T08:46:22.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2023-11-02T08:46:22.857 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T08:46:23.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672972 -lt 42949672972 2023-11-02T08:46:23.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:46:23.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509447 2023-11-02T08:46:23.263 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:46:23.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:46:23.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509447 2023-11-02T08:46:23.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:46:23.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509447 2023-11-02T08:46:23.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2023-11-02T08:46:23.265 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509447 2023-11-02T08:46:23.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:46:23.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509447 -lt 64424509447 2023-11-02T08:46:23.680 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T08:46:23.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:46:23.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:46:24.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2023-11-02T08:46:24.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:46:24.192 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:46:24.192 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:46:24.192 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:46:24.192 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:46:24.192 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:46:24.193 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:46:24.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T08:46:24.599 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:46:24.599 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:46:24.600 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:46:25.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2023-11-02T08:46:25.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T08:46:25.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T08:46:25.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: create_ec_pool: return 0 2023-11-02T08:46:25.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2023-11-02T08:46:25.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:46:25.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:46:25.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:46:25.627 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:46:25.627 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:46:25.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:46:25.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:46:25.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:46:25.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836506 2023-11-02T08:46:25.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836506 2023-11-02T08:46:25.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836506' 2023-11-02T08:46:25.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:46:25.747 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:46:25.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672975 2023-11-02T08:46:25.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672975 2023-11-02T08:46:25.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836506 1-42949672975' 2023-11-02T08:46:25.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:46:25.885 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:46:26.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509450 2023-11-02T08:46:26.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509450 2023-11-02T08:46:26.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836506 1-42949672975 2-64424509450' 2023-11-02T08:46:26.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:46:26.003 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836506 2023-11-02T08:46:26.003 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:46:26.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:46:26.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836506 2023-11-02T08:46:26.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:46:26.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836506 2023-11-02T08:46:26.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2023-11-02T08:46:26.006 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836506 2023-11-02T08:46:26.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:46:26.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836504 -lt 21474836506 2023-11-02T08:46:26.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T08:46:27.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T08:46:27.413 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:46:27.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836507 -lt 21474836506 2023-11-02T08:46:27.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:46:27.820 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672975 2023-11-02T08:46:27.820 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:46:27.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T08:46:27.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672975 2023-11-02T08:46:27.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:46:27.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672975 2023-11-02T08:46:27.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2023-11-02T08:46:27.823 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672975 2023-11-02T08:46:27.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T08:46:28.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672975 -lt 42949672975 2023-11-02T08:46:28.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:46:28.223 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509450 2023-11-02T08:46:28.223 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:46:28.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:46:28.224 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509450 2023-11-02T08:46:28.224 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:46:28.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509450 2023-11-02T08:46:28.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2023-11-02T08:46:28.226 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509450 2023-11-02T08:46:28.226 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:46:28.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509450 -lt 64424509450 2023-11-02T08:46:28.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2023-11-02T08:46:28.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T08:46:28.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T08:46:28.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T08:46:28.684 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-11-02T08:46:28.684 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T08:46:28.684 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T08:46:28.684 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T08:46:28.684 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T08:46:28.685 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T08:46:28.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-11-02T08:46:28.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T08:46:28.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T08:46:28.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T08:46:28.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:46:28.767 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:46:29.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:46:29.183 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:46:29.184 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:46:29.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:46:29.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:46:29.184 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:46:29.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836509 2023-11-02T08:46:29.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836509 2023-11-02T08:46:29.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836509' 2023-11-02T08:46:29.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:46:29.303 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:46:29.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672977 2023-11-02T08:46:29.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672977 2023-11-02T08:46:29.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836509 1-42949672977' 2023-11-02T08:46:29.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:46:29.422 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:46:29.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509453 2023-11-02T08:46:29.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509453 2023-11-02T08:46:29.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836509 1-42949672977 2-64424509453' 2023-11-02T08:46:29.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:46:29.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836509 2023-11-02T08:46:29.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:46:29.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:46:29.543 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836509 2023-11-02T08:46:29.543 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:46:29.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836509 2023-11-02T08:46:29.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2023-11-02T08:46:29.544 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836509 2023-11-02T08:46:29.545 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:46:29.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836509 -lt 21474836509 2023-11-02T08:46:29.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:46:29.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672977 2023-11-02T08:46:29.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:46:29.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T08:46:29.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672977 2023-11-02T08:46:29.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:46:29.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672977 2023-11-02T08:46:29.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2023-11-02T08:46:29.991 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672977 2023-11-02T08:46:29.991 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T08:46:30.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672977 -lt 42949672977 2023-11-02T08:46:30.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:46:30.396 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509453 2023-11-02T08:46:30.396 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:46:30.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:46:30.397 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509453 2023-11-02T08:46:30.397 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:46:30.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509453 2023-11-02T08:46:30.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2023-11-02T08:46:30.399 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509453 2023-11-02T08:46:30.399 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:46:30.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509453 -lt 64424509453 2023-11-02T08:46:30.776 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T08:46:30.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:46:30.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:46:31.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2023-11-02T08:46:31.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:46:31.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:46:31.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:46:31.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:46:31.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:46:31.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:46:31.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:46:31.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T08:46:31.687 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:46:31.687 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:46:31.688 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:46:32.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2023-11-02T08:46:32.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T08:46:32.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T08:46:32.193 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2023-11-02T08:46:32.193 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2023-11-02T08:46:32.602 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:46:32.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2023-11-02T08:46:32.613 INFO:tasks.workunit.client.0.smithi116.stdout:primary and soon to be divergent is 1 2023-11-02T08:46:32.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2023-11-02T08:46:32.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2023-11-02T08:46:32.965 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:46:32.966 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-02T08:46:19.402926+0000 0'0 29:12 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:46:18.062675+0000 0'0 2023-11-02T08:46:18.062675+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:32:42.597831+0000 0 0 2023-11-02T08:46:32.966 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:46:32.966 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:46:32.967 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:46:32.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2023-11-02T08:46:32.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2023-11-02T08:46:32.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2023-11-02T08:46:32.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2023-11-02T08:46:32.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2023-11-02T08:46:32.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2023-11-02T08:46:32.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2023-11-02T08:46:32.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2023-11-02T08:46:32.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2023-11-02T08:46:32.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2023-11-02T08:46:32.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2023-11-02T08:46:32.979 INFO:tasks.workunit.client.0.smithi116.stdout:writing initial objects 2023-11-02T08:46:32.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2023-11-02T08:46:32.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:32.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2023-11-02T08:46:33.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:33.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2023-11-02T08:46:33.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:33.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2023-11-02T08:46:33.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:33.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2023-11-02T08:46:33.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:33.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2023-11-02T08:46:34.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:34.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2023-11-02T08:46:34.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:34.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2023-11-02T08:46:34.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:34.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2023-11-02T08:46:34.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:34.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2023-11-02T08:46:34.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:34.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2023-11-02T08:46:35.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:35.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2023-11-02T08:46:35.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:35.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2023-11-02T08:46:35.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:35.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2023-11-02T08:46:35.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:35.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2023-11-02T08:46:35.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:35.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2023-11-02T08:46:36.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:36.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2023-11-02T08:46:36.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:36.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2023-11-02T08:46:36.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:36.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2023-11-02T08:46:36.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:36.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2023-11-02T08:46:37.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:37.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2023-11-02T08:46:37.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:37.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2023-11-02T08:46:37.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:37.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2023-11-02T08:46:37.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:37.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2023-11-02T08:46:37.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:37.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2023-11-02T08:46:38.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:38.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2023-11-02T08:46:38.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:38.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2023-11-02T08:46:38.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:38.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2023-11-02T08:46:38.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:38.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2023-11-02T08:46:38.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:38.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2023-11-02T08:46:39.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:39.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2023-11-02T08:46:39.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:39.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2023-11-02T08:46:39.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:39.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2023-11-02T08:46:39.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:39.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2023-11-02T08:46:39.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:39.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2023-11-02T08:46:40.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:40.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2023-11-02T08:46:40.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:40.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2023-11-02T08:46:40.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:40.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2023-11-02T08:46:40.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:40.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2023-11-02T08:46:41.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:41.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2023-11-02T08:46:41.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:41.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2023-11-02T08:46:41.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:41.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2023-11-02T08:46:41.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:41.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2023-11-02T08:46:41.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:41.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2023-11-02T08:46:42.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:42.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2023-11-02T08:46:42.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:42.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2023-11-02T08:46:42.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:42.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2023-11-02T08:46:42.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:42.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2023-11-02T08:46:43.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:43.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2023-11-02T08:46:43.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:43.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2023-11-02T08:46:43.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:43.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2023-11-02T08:46:43.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:43.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2023-11-02T08:46:43.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:43.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2023-11-02T08:46:44.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:44.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2023-11-02T08:46:44.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:44.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2023-11-02T08:46:44.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:44.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2023-11-02T08:46:44.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:44.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2023-11-02T08:46:44.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:44.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2023-11-02T08:46:45.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:45.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2023-11-02T08:46:45.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:45.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2023-11-02T08:46:45.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:45.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2023-11-02T08:46:45.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:45.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2023-11-02T08:46:45.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:45.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2023-11-02T08:46:46.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:46.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2023-11-02T08:46:46.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:46.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2023-11-02T08:46:46.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:46.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2023-11-02T08:46:46.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:46.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2023-11-02T08:46:47.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:47.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2023-11-02T08:46:47.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:47.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2023-11-02T08:46:47.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:47.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2023-11-02T08:46:47.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:47.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2023-11-02T08:46:47.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:47.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2023-11-02T08:46:48.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:48.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2023-11-02T08:46:48.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:48.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2023-11-02T08:46:48.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:48.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2023-11-02T08:46:48.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:48.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2023-11-02T08:46:49.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:49.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2023-11-02T08:46:49.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:49.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2023-11-02T08:46:49.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:49.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2023-11-02T08:46:49.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:49.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2023-11-02T08:46:49.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:49.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2023-11-02T08:46:50.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:50.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2023-11-02T08:46:50.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:50.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2023-11-02T08:46:50.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:50.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2023-11-02T08:46:50.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:50.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2023-11-02T08:46:50.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:50.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2023-11-02T08:46:51.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:51.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2023-11-02T08:46:51.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:51.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2023-11-02T08:46:51.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:51.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2023-11-02T08:46:51.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:51.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2023-11-02T08:46:52.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:52.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2023-11-02T08:46:52.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:52.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2023-11-02T08:46:52.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:52.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2023-11-02T08:46:52.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:52.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2023-11-02T08:46:52.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:52.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2023-11-02T08:46:53.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:53.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2023-11-02T08:46:53.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:53.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2023-11-02T08:46:53.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:53.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2023-11-02T08:46:53.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:53.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2023-11-02T08:46:53.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:53.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2023-11-02T08:46:54.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2023-11-02T08:46:54.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2023-11-02T08:46:54.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2023-11-02T08:46:54.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2023-11-02T08:46:54.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T08:46:54.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T08:46:54.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T08:46:54.413 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2023-11-02T08:46:54.414 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T08:46:54.414 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T08:46:54.414 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T08:46:54.415 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T08:46:54.415 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T08:46:54.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2023-11-02T08:46:54.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T08:46:54.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T08:46:54.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T08:46:54.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:46:54.471 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:46:54.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:46:54.850 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:46:54.851 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:46:54.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:46:54.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:46:54.851 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:46:54.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836516 2023-11-02T08:46:54.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836516 2023-11-02T08:46:54.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836516' 2023-11-02T08:46:54.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:46:54.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:46:55.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672984 2023-11-02T08:46:55.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672984 2023-11-02T08:46:55.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836516 1-42949672984' 2023-11-02T08:46:55.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:46:55.089 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:46:55.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509460 2023-11-02T08:46:55.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509460 2023-11-02T08:46:55.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836516 1-42949672984 2-64424509460' 2023-11-02T08:46:55.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:46:55.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836516 2023-11-02T08:46:55.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:46:55.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:46:55.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836516 2023-11-02T08:46:55.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:46:55.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836516 2023-11-02T08:46:55.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2023-11-02T08:46:55.210 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836516 2023-11-02T08:46:55.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:46:55.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836514 -lt 21474836516 2023-11-02T08:46:55.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T08:46:56.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T08:46:56.623 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:46:57.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836516 -lt 21474836516 2023-11-02T08:46:57.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:46:57.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672984 2023-11-02T08:46:57.042 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:46:57.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T08:46:57.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672984 2023-11-02T08:46:57.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:46:57.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672984 2023-11-02T08:46:57.044 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672984 2023-11-02T08:46:57.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2023-11-02T08:46:57.044 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T08:46:57.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672985 -lt 42949672984 2023-11-02T08:46:57.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:46:57.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509460 2023-11-02T08:46:57.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:46:57.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:46:57.460 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509460 2023-11-02T08:46:57.461 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:46:57.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509460 2023-11-02T08:46:57.462 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509460 2023-11-02T08:46:57.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2023-11-02T08:46:57.463 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:46:57.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509460 -lt 64424509460 2023-11-02T08:46:57.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T08:46:57.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:46:57.843 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:46:58.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2023-11-02T08:46:58.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:46:58.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:46:58.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:46:58.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:46:58.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:46:58.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:46:58.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:46:58.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T08:46:58.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:46:58.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:46:58.751 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:46:59.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2023-11-02T08:46:59.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T08:46:59.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T08:46:59.297 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2023-11-02T08:46:59.297 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2023-11-02T08:46:59.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2023-11-02T08:46:59.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2023-11-02T08:46:59.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2023-11-02T08:46:59.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2023-11-02T08:46:59.730 INFO:tasks.workunit.client.0.smithi116.stdout:blackholing osds 0 2 2023-11-02T08:46:59.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2023-11-02T08:46:59.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2023-11-02T08:47:00.113 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:47:00.113 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 0 0 0 48300 0 0 16 84 16 active+clean 2023-11-02T08:46:19.402926+0000 29'100 29:112 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:46:18.062675+0000 0'0 2023-11-02T08:46:18.062675+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:32:42.597831+0000 0 0 2023-11-02T08:47:00.113 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:47:00.113 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:47:00.114 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:47:00.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2023-11-02T08:47:00.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2023-11-02T08:47:00.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-11-02T08:47:00.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-11-02T08:47:00.125 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:47:00.125 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:47:00.126 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:47:00.126 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.0.asok 2023-11-02T08:47:00.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2023-11-02T08:47:00.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.0.asok config set objectstore_blackhole 1 2023-11-02T08:47:00.202 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T08:47:00.202 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2023-11-02T08:47:00.202 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T08:47:00.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2023-11-02T08:47:00.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2023-11-02T08:47:00.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-11-02T08:47:00.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-11-02T08:47:00.210 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:47:00.210 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:47:00.210 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:47:00.211 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.2.asok 2023-11-02T08:47:00.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2023-11-02T08:47:00.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.2.asok config set objectstore_blackhole 1 2023-11-02T08:47:00.287 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T08:47:00.287 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2023-11-02T08:47:00.287 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T08:47:00.294 INFO:tasks.workunit.client.0.smithi116.stdout:writing divergent object 2023-11-02T08:47:00.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2023-11-02T08:47:00.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2023-11-02T08:47:00.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2023-11-02T08:47:01.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2023-11-02T08:47:01.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2023-11-02T08:47:01.481 INFO:tasks.workunit.client.0.smithi116.stdout:created pool test snap snap1 2023-11-02T08:47:01.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2023-11-02T08:47:01.485 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2023-11-02T08:47:01.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2023-11-02T08:47:11.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2023-11-02T08:47:11.488 INFO:tasks.workunit.client.0.smithi116.stderr:rados: no process found 2023-11-02T08:47:11.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2023-11-02T08:47:11.489 INFO:tasks.workunit.client.0.smithi116.stdout:killing all the osds 2023-11-02T08:47:11.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2023-11-02T08:47:11.854 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:47:11.854 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 101 0 0 0 0 48783 0 0 14 90 14 active+clean 2023-11-02T08:46:19.402926+0000 30'104 30:116 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:46:18.062675+0000 0'0 2023-11-02T08:46:18.062675+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:32:42.597831+0000 0 0 2023-11-02T08:47:11.855 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:47:11.855 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:47:11.855 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:47:11.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2023-11-02T08:47:11.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:47:11.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:47:11.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:47:11.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:47:11.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:47:11.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:47:11.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2023-11-02T08:47:11.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2023-11-02T08:47:12.472 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 is already down. 2023-11-02T08:47:12.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2023-11-02T08:47:12.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2023-11-02T08:47:13.210 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 is already down. 2023-11-02T08:47:13.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2023-11-02T08:47:13.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2023-11-02T08:47:13.740 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 is already down. 2023-11-02T08:47:13.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2023-11-02T08:47:13.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2023-11-02T08:47:14.288 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 is already out. 2023-11-02T08:47:14.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2023-11-02T08:47:14.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2023-11-02T08:47:14.806 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 is already out. 2023-11-02T08:47:14.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2023-11-02T08:47:14.818 INFO:tasks.workunit.client.0.smithi116.stdout:bringing up non_divergent 0 2 2023-11-02T08:47:14.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2023-11-02T08:47:15.176 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:47:15.176 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 101 0 0 0 0 48783 0 0 14 90 14 stale+active+clean 2023-11-02T08:46:19.402926+0000 30'104 30:116 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:46:18.062675+0000 0'0 2023-11-02T08:46:18.062675+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:32:42.597831+0000 0 0 2023-11-02T08:47:15.177 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:47:15.177 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:47:15.177 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:47:15.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2023-11-02T08:47:15.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2023-11-02T08:47:15.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2023-11-02T08:47:15.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:47:15.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-11-02T08:47:15.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:47:15.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2023-11-02T08:47:15.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:47:15.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:47:15.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:47:15.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:47:15.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2023-11-02T08:47:15.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2023-11-02T08:47:15.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:47:15.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:47:15.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:47:15.193 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:47:15.194 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:47:15.194 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:47:15.194 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:47:15.195 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:47:15.195 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:47:15.195 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:47:15.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:47:15.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:47:15.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:47:15.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:47:15.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:47:15.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:47:15.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:47:15.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:47:15.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:47:15.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:47:15.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2023-11-02T08:47:15.198 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T08:47:15.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-11-02T08:47:15.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:47:15.199 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2023-11-02T08:47:15.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-11-02T08:47:15.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:47:15.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:47:15.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:47:15.223 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:47:15.220+0000 7fd0d1a69500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:47:15.224 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:47:15.222+0000 7fd0d1a69500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:47:15.225 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:47:15.223+0000 7fd0d1a69500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:47:15.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-11-02T08:47:15.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:47:15.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:47:15.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:47:15.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:47:15.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:47:15.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:47:15.608 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:47:15.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:47:15.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:47:15.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:47:15.738+0000 7fd0d1a69500 -1 Falling back to public interface 2023-11-02T08:47:15.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:47:16.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:47:16.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:47:16.996 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:47:16.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:47:16.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:47:16.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:47:17.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:47:18.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:47:18.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:47:18.384 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:47:18.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:47:18.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:47:18.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:47:18.526 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:47:18.523+0000 7fd0d1a69500 -1 osd.0 29 log_to_monitors true 2023-11-02T08:47:18.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:47:19.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:47:19.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:47:19.802 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:47:19.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:47:19.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:47:19.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:47:20.189 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/135069,v1:127.0.0.1:6803/135069] [v2:127.0.0.1:6804/135069,v1:127.0.0.1:6805/135069] exists,up 5f38bac9-3e89-4df4-9a41-799f865c97c8 2023-11-02T08:47:20.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:47:20.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:47:20.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:47:20.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2023-11-02T08:47:20.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2023-11-02T08:47:20.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2023-11-02T08:47:20.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:47:20.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2023-11-02T08:47:20.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:47:20.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2023-11-02T08:47:20.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:47:20.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:47:20.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:47:20.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:47:20.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2023-11-02T08:47:20.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2023-11-02T08:47:20.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:47:20.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:47:20.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:47:20.197 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:47:20.197 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:47:20.198 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:47:20.198 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:47:20.198 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:47:20.199 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:47:20.199 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:47:20.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:47:20.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:47:20.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:47:20.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:47:20.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:47:20.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:47:20.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:47:20.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:47:20.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:47:20.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:47:20.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2023-11-02T08:47:20.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2023-11-02T08:47:20.202 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T08:47:20.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:47:20.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2023-11-02T08:47:20.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2023-11-02T08:47:20.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:47:20.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:47:20.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:47:20.225 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:47:20.222+0000 7fa041307500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:47:20.227 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:47:20.224+0000 7fa041307500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:47:20.228 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:47:20.225+0000 7fa041307500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:47:20.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2023-11-02T08:47:20.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:47:20.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T08:47:20.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:47:20.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:47:20.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:47:20.637 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:47:20.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:47:20.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:47:20.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:47:20.752 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:47:20.749+0000 7fa041307500 -1 Falling back to public interface 2023-11-02T08:47:21.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:47:22.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:47:22.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:47:22.055 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:47:22.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:47:22.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:47:22.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:47:22.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:47:23.370 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:47:23.367+0000 7fa041307500 -1 osd.2 29 log_to_monitors true 2023-11-02T08:47:23.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:47:23.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:47:23.442 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:47:23.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:47:23.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:47:23.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:47:23.880 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up out weight 0 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/135638,v1:127.0.0.1:6811/135638] [v2:127.0.0.1:6812/135638,v1:127.0.0.1:6813/135638] exists,up c3024fa8-7c9d-4674-b3d4-e8b9122765a2 2023-11-02T08:47:23.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:47:23.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:47:23.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:47:23.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2023-11-02T08:47:23.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2023-11-02T08:47:24.429 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 is already in. 2023-11-02T08:47:24.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2023-11-02T08:47:24.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2023-11-02T08:47:25.215 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 is already in. 2023-11-02T08:47:25.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2023-11-02T08:47:30.230 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2023-11-02T08:47:30.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2023-11-02T08:47:30.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2023-11-02T08:47:30.231 INFO:tasks.workunit.client.0.smithi116.stdout:writing non-divergent object existing_4 2023-11-02T08:47:30.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2023-11-02T08:47:30.598 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:47:30.598 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2023-11-02T08:47:25.405503+0000 29'100 43:124 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2023-11-02T08:46:18.062675+0000 0'0 2023-11-02T08:46:18.062675+0000 0 0 periodic scrub scheduled @ 2023-11-03T18:42:03.214734+0000 0 0 2023-11-02T08:47:30.598 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:47:30.598 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:47:30.599 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:47:30.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2023-11-02T08:47:30.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2023-11-02T08:47:30.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2023-11-02T08:47:30.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T08:47:30.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T08:47:30.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T08:47:30.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2023-11-02T08:47:30.715 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T08:47:30.715 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T08:47:30.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T08:47:30.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T08:47:30.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T08:47:30.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2023-11-02T08:47:30.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T08:47:30.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T08:47:30.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T08:47:30.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:47:30.766 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:47:31.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:47:31.218 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:47:31.219 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:47:31.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:47:31.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:47:31.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:47:31.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=158913789957 2023-11-02T08:47:31.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 158913789957 2023-11-02T08:47:31.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-158913789957' 2023-11-02T08:47:31.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:47:31.338 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:47:31.424 INFO:tasks.workunit.client.0.smithi116.stderr:Error ENXIO: problem getting command descriptions from osd.1 2023-11-02T08:47:31.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2023-11-02T08:47:31.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2023-11-02T08:47:31.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2023-11-02T08:47:31.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:47:31.426 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:47:31.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=167503724548 2023-11-02T08:47:31.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 167503724548 2023-11-02T08:47:31.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-158913789957 2-167503724548' 2023-11-02T08:47:31.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:47:31.546 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-158913789957 2023-11-02T08:47:31.546 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:47:31.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:47:31.547 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-158913789957 2023-11-02T08:47:31.547 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:47:31.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=158913789957 2023-11-02T08:47:31.548 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 158913789957 2023-11-02T08:47:31.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 158913789957' 2023-11-02T08:47:31.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:47:31.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 158913789957 -lt 158913789957 2023-11-02T08:47:31.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:47:31.928 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-167503724548 2023-11-02T08:47:31.928 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:47:31.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:47:31.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-167503724548 2023-11-02T08:47:31.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:47:31.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=167503724548 2023-11-02T08:47:31.930 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 167503724548 2023-11-02T08:47:31.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 167503724548' 2023-11-02T08:47:31.931 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:47:32.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 167503724548 -lt 167503724548 2023-11-02T08:47:32.345 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T08:47:32.345 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:47:32.346 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:47:32.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2023-11-02T08:47:32.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:47:32.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:47:32.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:47:32.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:47:32.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:47:32.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:47:32.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:47:33.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:47:33.215 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:47:33.215 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:47:33.215 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:47:33.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:47:33.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2023-11-02T08:47:33.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2023-11-02T08:47:33.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2023-11-02T08:47:33.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:47:33.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T08:47:33.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:47:33.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:47:33.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:47:33.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:47:33.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:47:33.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:47:33.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:47:33.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:47:34.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:47:34.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:47:34.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:47:34.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:47:34.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:47:34.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:47:34.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:47:34.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:47:34.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:47:34.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:47:34.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:47:34.694 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:47:34.695 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:47:35.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:47:35.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:47:35.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2023-11-02T08:47:35.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:47:35.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2023-11-02T08:47:35.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:47:35.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:47:35.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:47:35.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:47:35.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:47:35.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:47:35.437 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:47:35.437 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:47:35.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:47:35.805 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:47:35.805 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:47:35.805 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:47:36.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:47:36.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:47:36.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:47:36.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:47:36.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:47:36.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:47:36.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:47:36.310 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:47:36.310 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:47:36.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:47:36.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:47:36.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2023-11-02T08:47:36.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:47:36.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2023-11-02T08:47:37.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:47:37.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:47:37.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:47:37.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:47:37.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:47:37.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:47:37.217 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:47:37.217 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:47:37.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:47:37.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:47:37.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:47:37.583 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:47:38.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:47:38.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:47:38.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:47:38.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:47:38.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:47:38.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:47:38.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:47:38.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:47:38.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:47:38.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:47:38.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:47:38.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2023-11-02T08:47:38.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:47:38.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2023-11-02T08:47:39.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:47:39.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:47:39.400 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:47:39.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:47:39.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:47:39.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:47:39.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:47:39.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:47:39.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:47:39.774 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:47:39.774 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:47:39.774 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:47:40.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:47:40.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:47:40.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:47:40.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:47:40.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:47:40.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:47:40.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:47:40.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:47:40.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:47:40.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:47:40.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:47:40.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2023-11-02T08:47:40.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:47:40.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2023-11-02T08:47:42.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:47:42.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:47:42.431 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:47:42.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:47:42.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:47:42.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:47:42.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:47:42.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:47:42.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:47:42.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:47:42.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:47:42.812 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:47:43.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:47:43.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:47:43.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:47:43.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:47:43.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:47:43.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:47:43.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:47:43.319 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:47:43.319 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:47:43.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:47:43.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:47:43.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2023-11-02T08:47:43.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:47:43.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2023-11-02T08:47:47.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:47:47.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:47:47.065 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:47:47.065 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:47:47.065 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:47:47.066 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:47:47.066 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:47:47.066 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:47:47.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:47:47.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:47:47.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:47:47.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:47:47.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:47:47.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:47:47.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:47:47.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:47:47.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:47:47.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:47:47.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:47:47.984 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:47:47.984 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:47:48.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:47:48.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:47:48.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2023-11-02T08:47:48.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:47:48.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2023-11-02T08:47:54.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:47:54.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:47:54.894 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:47:54.894 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:47:54.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:47:54.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:47:54.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:47:54.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:47:55.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:47:55.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:47:55.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:47:55.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:47:55.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:47:55.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:47:55.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:47:55.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:47:55.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:47:55.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:47:55.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:47:55.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:47:55.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:47:56.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:47:56.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:47:56.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2023-11-02T08:47:56.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:47:56.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2023-11-02T08:48:03.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:48:03.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:48:03.603 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:48:03.603 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:48:03.603 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:48:03.603 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:48:03.603 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:48:03.604 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:48:03.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:48:03.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:48:03.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:48:03.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:48:04.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:48:04.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:48:04.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:48:04.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:48:04.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:48:04.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:48:04.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:48:04.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:48:04.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:48:05.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:48:05.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:48:05.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2023-11-02T08:48:05.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2023-11-02T08:48:05.589 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T08:48:05.589 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_fingerprint": "2a860d17-8260-45c6-94e5-d9c79ce24314", 2023-11-02T08:48:05.589 INFO:tasks.workunit.client.0.smithi116.stdout: "version": "18.2.0-1159-gb5a9486b", 2023-11-02T08:48:05.590 INFO:tasks.workunit.client.0.smithi116.stdout: "commit": "b5a9486b4a6f144f83c51b7ec33111a2c86dc081", 2023-11-02T08:48:05.590 INFO:tasks.workunit.client.0.smithi116.stdout: "timestamp": "2023-11-02T08:48:05.586085+0000", 2023-11-02T08:48:05.590 INFO:tasks.workunit.client.0.smithi116.stdout: "tag": "", 2023-11-02T08:48:05.590 INFO:tasks.workunit.client.0.smithi116.stdout: "health": { 2023-11-02T08:48:05.591 INFO:tasks.workunit.client.0.smithi116.stdout: "status": "HEALTH_WARN", 2023-11-02T08:48:05.591 INFO:tasks.workunit.client.0.smithi116.stdout: "checks": { 2023-11-02T08:48:05.591 INFO:tasks.workunit.client.0.smithi116.stdout: "OSDMAP_FLAGS": { 2023-11-02T08:48:05.591 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:48:05.591 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:48:05.591 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "nodown,noout,noin flag(s) set", 2023-11-02T08:48:05.592 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 17 2023-11-02T08:48:05.592 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.592 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [], 2023-11-02T08:48:05.592 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:48:05.592 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.593 INFO:tasks.workunit.client.0.smithi116.stdout: "OSD_DOWN": { 2023-11-02T08:48:05.593 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:48:05.593 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:48:05.593 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "1 osds down", 2023-11-02T08:48:05.593 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 1 2023-11-02T08:48:05.594 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.594 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:48:05.594 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.594 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "osd.1 (root=default,host=smithi116) is down" 2023-11-02T08:48:05.594 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.594 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:05.595 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:48:05.595 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.595 INFO:tasks.workunit.client.0.smithi116.stdout: "PG_DEGRADED": { 2023-11-02T08:48:05.595 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:48:05.595 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:48:05.596 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2023-11-02T08:48:05.596 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 1 2023-11-02T08:48:05.596 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.596 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:48:05.596 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.597 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2023-11-02T08:48:05.597 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.597 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:05.597 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:48:05.597 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.597 INFO:tasks.workunit.client.0.smithi116.stdout: "POOL_APP_NOT_ENABLED": { 2023-11-02T08:48:05.598 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:48:05.598 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:48:05.598 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "1 pool(s) do not have an application enabled", 2023-11-02T08:48:05.598 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 1 2023-11-02T08:48:05.598 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.599 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:48:05.599 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.599 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "application not enabled on pool 'test'" 2023-11-02T08:48:05.599 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.599 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.600 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2023-11-02T08:48:05.600 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.600 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:05.600 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:48:05.600 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.600 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.601 INFO:tasks.workunit.client.0.smithi116.stdout: "mutes": [] 2023-11-02T08:48:05.601 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.601 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap_first_committed": 1, 2023-11-02T08:48:05.601 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap_last_committed": 1, 2023-11-02T08:48:05.601 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap": { 2023-11-02T08:48:05.602 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 1, 2023-11-02T08:48:05.602 INFO:tasks.workunit.client.0.smithi116.stdout: "fsid": "0c9a372f-cec7-47d9-a309-8248453aabeb", 2023-11-02T08:48:05.602 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:44:34.120499Z", 2023-11-02T08:48:05.602 INFO:tasks.workunit.client.0.smithi116.stdout: "created": "2023-11-02T08:44:34.120499Z", 2023-11-02T08:48:05.602 INFO:tasks.workunit.client.0.smithi116.stdout: "min_mon_release": 18, 2023-11-02T08:48:05.602 INFO:tasks.workunit.client.0.smithi116.stdout: "min_mon_release_name": "reef", 2023-11-02T08:48:05.603 INFO:tasks.workunit.client.0.smithi116.stdout: "election_strategy": 1, 2023-11-02T08:48:05.603 INFO:tasks.workunit.client.0.smithi116.stdout: "disallowed_leaders: ": "", 2023-11-02T08:48:05.603 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode": false, 2023-11-02T08:48:05.603 INFO:tasks.workunit.client.0.smithi116.stdout: "tiebreaker_mon": "", 2023-11-02T08:48:05.603 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_ranks: ": "", 2023-11-02T08:48:05.605 INFO:tasks.workunit.client.0.smithi116.stdout: "features": { 2023-11-02T08:48:05.605 INFO:tasks.workunit.client.0.smithi116.stdout: "persistent": [ 2023-11-02T08:48:05.605 INFO:tasks.workunit.client.0.smithi116.stdout: "kraken", 2023-11-02T08:48:05.605 INFO:tasks.workunit.client.0.smithi116.stdout: "luminous", 2023-11-02T08:48:05.605 INFO:tasks.workunit.client.0.smithi116.stdout: "mimic", 2023-11-02T08:48:05.606 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap-prune", 2023-11-02T08:48:05.606 INFO:tasks.workunit.client.0.smithi116.stdout: "nautilus", 2023-11-02T08:48:05.606 INFO:tasks.workunit.client.0.smithi116.stdout: "octopus", 2023-11-02T08:48:05.606 INFO:tasks.workunit.client.0.smithi116.stdout: "pacific", 2023-11-02T08:48:05.606 INFO:tasks.workunit.client.0.smithi116.stdout: "elector-pinging", 2023-11-02T08:48:05.607 INFO:tasks.workunit.client.0.smithi116.stdout: "quincy", 2023-11-02T08:48:05.607 INFO:tasks.workunit.client.0.smithi116.stdout: "reef" 2023-11-02T08:48:05.607 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:05.607 INFO:tasks.workunit.client.0.smithi116.stdout: "optional": [] 2023-11-02T08:48:05.607 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.607 INFO:tasks.workunit.client.0.smithi116.stdout: "mons": [ 2023-11-02T08:48:05.608 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.608 INFO:tasks.workunit.client.0.smithi116.stdout: "rank": 0, 2023-11-02T08:48:05.608 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "a", 2023-11-02T08:48:05.608 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:48:05.608 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:05.609 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.609 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:05.609 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:7115", 2023-11-02T08:48:05.609 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 0 2023-11-02T08:48:05.609 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.609 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.610 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.610 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:7115/0", 2023-11-02T08:48:05.610 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:7115/0", 2023-11-02T08:48:05.610 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 0, 2023-11-02T08:48:05.610 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 0, 2023-11-02T08:48:05.611 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_location": "{}" 2023-11-02T08:48:05.611 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.611 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.611 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.611 INFO:tasks.workunit.client.0.smithi116.stdout: "quorum": [ 2023-11-02T08:48:05.611 INFO:tasks.workunit.client.0.smithi116.stdout: 0 2023-11-02T08:48:05.612 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:05.612 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap": { 2023-11-02T08:48:05.612 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 43, 2023-11-02T08:48:05.612 INFO:tasks.workunit.client.0.smithi116.stdout: "fsid": "0c9a372f-cec7-47d9-a309-8248453aabeb", 2023-11-02T08:48:05.612 INFO:tasks.workunit.client.0.smithi116.stdout: "created": "2023-11-02T08:44:34.942253+0000", 2023-11-02T08:48:05.613 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:47:25.002267+0000", 2023-11-02T08:48:05.613 INFO:tasks.workunit.client.0.smithi116.stdout: "last_up_change": "2023-11-02T08:47:23.571419+0000", 2023-11-02T08:48:05.613 INFO:tasks.workunit.client.0.smithi116.stdout: "last_in_change": "2023-11-02T08:47:24.817974+0000", 2023-11-02T08:48:05.613 INFO:tasks.workunit.client.0.smithi116.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-11-02T08:48:05.613 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_num": 5800384, 2023-11-02T08:48:05.613 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_set": [ 2023-11-02T08:48:05.614 INFO:tasks.workunit.client.0.smithi116.stdout: "nodown", 2023-11-02T08:48:05.614 INFO:tasks.workunit.client.0.smithi116.stdout: "noin", 2023-11-02T08:48:05.614 INFO:tasks.workunit.client.0.smithi116.stdout: "noout", 2023-11-02T08:48:05.614 INFO:tasks.workunit.client.0.smithi116.stdout: "pglog_hardlimit", 2023-11-02T08:48:05.614 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snapdirs", 2023-11-02T08:48:05.615 INFO:tasks.workunit.client.0.smithi116.stdout: "recovery_deletes", 2023-11-02T08:48:05.615 INFO:tasks.workunit.client.0.smithi116.stdout: "sortbitwise" 2023-11-02T08:48:05.615 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:05.615 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_version": 10, 2023-11-02T08:48:05.615 INFO:tasks.workunit.client.0.smithi116.stdout: "full_ratio": 0.99000000953674316, 2023-11-02T08:48:05.615 INFO:tasks.workunit.client.0.smithi116.stdout: "backfillfull_ratio": 0.99000000953674316, 2023-11-02T08:48:05.616 INFO:tasks.workunit.client.0.smithi116.stdout: "nearfull_ratio": 0.85000002384185791, 2023-11-02T08:48:05.616 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_snapshot": "", 2023-11-02T08:48:05.616 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_max": 1, 2023-11-02T08:48:05.616 INFO:tasks.workunit.client.0.smithi116.stdout: "max_osd": 3, 2023-11-02T08:48:05.616 INFO:tasks.workunit.client.0.smithi116.stdout: "require_min_compat_client": "luminous", 2023-11-02T08:48:05.617 INFO:tasks.workunit.client.0.smithi116.stdout: "min_compat_client": "jewel", 2023-11-02T08:48:05.617 INFO:tasks.workunit.client.0.smithi116.stdout: "require_osd_release": "reef", 2023-11-02T08:48:05.617 INFO:tasks.workunit.client.0.smithi116.stdout: "allow_crimson": false, 2023-11-02T08:48:05.617 INFO:tasks.workunit.client.0.smithi116.stdout: "pools": [ 2023-11-02T08:48:05.617 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.617 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": 1, 2023-11-02T08:48:05.618 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_name": "test", 2023-11-02T08:48:05.618 INFO:tasks.workunit.client.0.smithi116.stdout: "create_time": "2023-11-02T08:46:17.984400+0000", 2023-11-02T08:48:05.618 INFO:tasks.workunit.client.0.smithi116.stdout: "flags": 16389, 2023-11-02T08:48:05.620 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2023-11-02T08:48:05.620 INFO:tasks.workunit.client.0.smithi116.stdout: "type": 3, 2023-11-02T08:48:05.621 INFO:tasks.workunit.client.0.smithi116.stdout: "size": 3, 2023-11-02T08:48:05.621 INFO:tasks.workunit.client.0.smithi116.stdout: "min_size": 2, 2023-11-02T08:48:05.621 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_rule": 1, 2023-11-02T08:48:05.622 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_count": 0, 2023-11-02T08:48:05.622 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_target": 0, 2023-11-02T08:48:05.622 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_barrier": 0, 2023-11-02T08:48:05.624 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-11-02T08:48:05.624 INFO:tasks.workunit.client.0.smithi116.stdout: "object_hash": 2, 2023-11-02T08:48:05.624 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_autoscale_mode": "off", 2023-11-02T08:48:05.624 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num": 1, 2023-11-02T08:48:05.624 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_placement_num": 1, 2023-11-02T08:48:05.625 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_placement_num_target": 1, 2023-11-02T08:48:05.625 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num_target": 1, 2023-11-02T08:48:05.625 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num_pending": 1, 2023-11-02T08:48:05.625 INFO:tasks.workunit.client.0.smithi116.stdout: "last_pg_merge_meta": { 2023-11-02T08:48:05.626 INFO:tasks.workunit.client.0.smithi116.stdout: "source_pgid": "0.0", 2023-11-02T08:48:05.626 INFO:tasks.workunit.client.0.smithi116.stdout: "ready_epoch": 0, 2023-11-02T08:48:05.626 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_started": 0, 2023-11-02T08:48:05.627 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_clean": 0, 2023-11-02T08:48:05.627 INFO:tasks.workunit.client.0.smithi116.stdout: "source_version": "0'0", 2023-11-02T08:48:05.627 INFO:tasks.workunit.client.0.smithi116.stdout: "target_version": "0'0" 2023-11-02T08:48:05.627 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.627 INFO:tasks.workunit.client.0.smithi116.stdout: "last_change": "30", 2023-11-02T08:48:05.628 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend": "0", 2023-11-02T08:48:05.628 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend_prenautilus": "0", 2023-11-02T08:48:05.628 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend_preluminous": "0", 2023-11-02T08:48:05.628 INFO:tasks.workunit.client.0.smithi116.stdout: "auid": 0, 2023-11-02T08:48:05.629 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_mode": "pool", 2023-11-02T08:48:05.629 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_seq": 1, 2023-11-02T08:48:05.629 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_epoch": 30, 2023-11-02T08:48:05.629 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_snaps": [ 2023-11-02T08:48:05.630 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.630 INFO:tasks.workunit.client.0.smithi116.stdout: "snapid": 1, 2023-11-02T08:48:05.630 INFO:tasks.workunit.client.0.smithi116.stdout: "stamp": "2023-11-02T08:47:01.329439+0000", 2023-11-02T08:48:05.630 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "snap1" 2023-11-02T08:48:05.630 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.631 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:05.632 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_snaps": "[]", 2023-11-02T08:48:05.632 INFO:tasks.workunit.client.0.smithi116.stdout: "quota_max_bytes": 0, 2023-11-02T08:48:05.632 INFO:tasks.workunit.client.0.smithi116.stdout: "quota_max_objects": 0, 2023-11-02T08:48:05.632 INFO:tasks.workunit.client.0.smithi116.stdout: "tiers": [], 2023-11-02T08:48:05.633 INFO:tasks.workunit.client.0.smithi116.stdout: "tier_of": -1, 2023-11-02T08:48:05.633 INFO:tasks.workunit.client.0.smithi116.stdout: "read_tier": -1, 2023-11-02T08:48:05.633 INFO:tasks.workunit.client.0.smithi116.stdout: "write_tier": -1, 2023-11-02T08:48:05.633 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_mode": "none", 2023-11-02T08:48:05.633 INFO:tasks.workunit.client.0.smithi116.stdout: "target_max_bytes": 0, 2023-11-02T08:48:05.634 INFO:tasks.workunit.client.0.smithi116.stdout: "target_max_objects": 0, 2023-11-02T08:48:05.634 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-11-02T08:48:05.634 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-11-02T08:48:05.634 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_full_ratio_micro": 800000, 2023-11-02T08:48:05.634 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_min_flush_age": 0, 2023-11-02T08:48:05.635 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_min_evict_age": 0, 2023-11-02T08:48:05.635 INFO:tasks.workunit.client.0.smithi116.stdout: "erasure_code_profile": "myprofile", 2023-11-02T08:48:05.635 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_params": { 2023-11-02T08:48:05.635 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "none" 2023-11-02T08:48:05.635 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.636 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_period": 0, 2023-11-02T08:48:05.636 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_count": 0, 2023-11-02T08:48:05.636 INFO:tasks.workunit.client.0.smithi116.stdout: "use_gmt_hitset": true, 2023-11-02T08:48:05.636 INFO:tasks.workunit.client.0.smithi116.stdout: "min_read_recency_for_promote": 0, 2023-11-02T08:48:05.637 INFO:tasks.workunit.client.0.smithi116.stdout: "min_write_recency_for_promote": 0, 2023-11-02T08:48:05.637 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_grade_decay_rate": 0, 2023-11-02T08:48:05.637 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_search_last_n": 0, 2023-11-02T08:48:05.637 INFO:tasks.workunit.client.0.smithi116.stdout: "grade_table": [], 2023-11-02T08:48:05.638 INFO:tasks.workunit.client.0.smithi116.stdout: "stripe_width": 8192, 2023-11-02T08:48:05.638 INFO:tasks.workunit.client.0.smithi116.stdout: "expected_num_objects": 0, 2023-11-02T08:48:05.638 INFO:tasks.workunit.client.0.smithi116.stdout: "fast_read": false, 2023-11-02T08:48:05.638 INFO:tasks.workunit.client.0.smithi116.stdout: "options": {}, 2023-11-02T08:48:05.639 INFO:tasks.workunit.client.0.smithi116.stdout: "application_metadata": {} 2023-11-02T08:48:05.639 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.640 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:05.640 INFO:tasks.workunit.client.0.smithi116.stdout: "osds": [ 2023-11-02T08:48:05.640 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.640 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:48:05.640 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "5f38bac9-3e89-4df4-9a41-799f865c97c8", 2023-11-02T08:48:05.641 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 1, 2023-11-02T08:48:05.641 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:48:05.641 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:48:05.641 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:48:05.641 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 5, 2023-11-02T08:48:05.642 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 29, 2023-11-02T08:48:05.642 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 37, 2023-11-02T08:48:05.643 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 42, 2023-11-02T08:48:05.643 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 31, 2023-11-02T08:48:05.643 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:48:05.643 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:48:05.643 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:05.644 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.644 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:05.644 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6802", 2023-11-02T08:48:05.644 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 135069 2023-11-02T08:48:05.644 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.644 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.645 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:05.645 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6803", 2023-11-02T08:48:05.645 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 135069 2023-11-02T08:48:05.645 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.645 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.646 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.646 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:48:05.646 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:05.646 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.646 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:05.647 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6804", 2023-11-02T08:48:05.647 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 135069 2023-11-02T08:48:05.647 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.647 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.647 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:05.648 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6805", 2023-11-02T08:48:05.648 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 135069 2023-11-02T08:48:05.648 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.648 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.648 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.648 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:48:05.649 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:05.649 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.649 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:05.649 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6808", 2023-11-02T08:48:05.649 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 135069 2023-11-02T08:48:05.650 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.650 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.650 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:05.650 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6809", 2023-11-02T08:48:05.650 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 135069 2023-11-02T08:48:05.651 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.651 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.651 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.651 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:48:05.651 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:05.651 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.652 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:05.652 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6806", 2023-11-02T08:48:05.652 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 135069 2023-11-02T08:48:05.652 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.652 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.653 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:05.653 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6807", 2023-11-02T08:48:05.653 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 135069 2023-11-02T08:48:05.653 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.653 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.654 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.654 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6803/135069", 2023-11-02T08:48:05.654 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6805/135069", 2023-11-02T08:48:05.654 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6809/135069", 2023-11-02T08:48:05.654 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6807/135069", 2023-11-02T08:48:05.654 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:48:05.655 INFO:tasks.workunit.client.0.smithi116.stdout: "exists", 2023-11-02T08:48:05.655 INFO:tasks.workunit.client.0.smithi116.stdout: "up" 2023-11-02T08:48:05.655 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.655 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.655 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.656 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:48:05.656 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "00de077e-ecc3-49b3-b8fc-83cfdd948349", 2023-11-02T08:48:05.656 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 0, 2023-11-02T08:48:05.656 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:48:05.656 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:48:05.657 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:48:05.657 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 0, 2023-11-02T08:48:05.657 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 0, 2023-11-02T08:48:05.657 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 10, 2023-11-02T08:48:05.657 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 25, 2023-11-02T08:48:05.657 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 32, 2023-11-02T08:48:05.658 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:48:05.658 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:48:05.658 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:05.658 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.658 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:05.659 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6810", 2023-11-02T08:48:05.659 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 128752 2023-11-02T08:48:05.659 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.659 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.659 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:05.660 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6811", 2023-11-02T08:48:05.660 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 128752 2023-11-02T08:48:05.660 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.660 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.660 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.661 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:48:05.661 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:05.661 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.661 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:05.661 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6812", 2023-11-02T08:48:05.661 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 128752 2023-11-02T08:48:05.662 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.662 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.662 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:05.662 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6813", 2023-11-02T08:48:05.662 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 128752 2023-11-02T08:48:05.663 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.663 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.663 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.663 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:48:05.663 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:05.664 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.664 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:05.664 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6816", 2023-11-02T08:48:05.664 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 128752 2023-11-02T08:48:05.664 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.664 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.665 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:05.665 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6817", 2023-11-02T08:48:05.665 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 128752 2023-11-02T08:48:05.665 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.665 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.666 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.666 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:48:05.666 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:05.666 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.667 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:05.667 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6814", 2023-11-02T08:48:05.667 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 128752 2023-11-02T08:48:05.667 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.667 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.668 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:05.668 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6815", 2023-11-02T08:48:05.668 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 128752 2023-11-02T08:48:05.668 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.668 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.668 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.669 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6811/128752", 2023-11-02T08:48:05.669 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6813/128752", 2023-11-02T08:48:05.669 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6817/128752", 2023-11-02T08:48:05.669 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6815/128752", 2023-11-02T08:48:05.669 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:48:05.670 INFO:tasks.workunit.client.0.smithi116.stdout: "exists" 2023-11-02T08:48:05.670 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.670 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.670 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.670 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:48:05.671 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "c3024fa8-7c9d-4674-b3d4-e8b9122765a2", 2023-11-02T08:48:05.671 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 1, 2023-11-02T08:48:05.671 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:48:05.671 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:48:05.671 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:48:05.671 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 15, 2023-11-02T08:48:05.672 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 29, 2023-11-02T08:48:05.672 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 39, 2023-11-02T08:48:05.672 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 0, 2023-11-02T08:48:05.672 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 33, 2023-11-02T08:48:05.672 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:48:05.673 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:48:05.673 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:05.673 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.673 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:05.673 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6810", 2023-11-02T08:48:05.674 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 135638 2023-11-02T08:48:05.674 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.674 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.674 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:05.674 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6811", 2023-11-02T08:48:05.674 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 135638 2023-11-02T08:48:05.675 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.675 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.675 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.675 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:48:05.675 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:05.676 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.676 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:05.676 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6812", 2023-11-02T08:48:05.676 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 135638 2023-11-02T08:48:05.676 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.676 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.677 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:05.677 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6813", 2023-11-02T08:48:05.677 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 135638 2023-11-02T08:48:05.677 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.677 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.678 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.678 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:48:05.678 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:05.678 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.678 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:05.679 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6816", 2023-11-02T08:48:05.679 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 135638 2023-11-02T08:48:05.679 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.679 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.679 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:05.679 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6817", 2023-11-02T08:48:05.680 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 135638 2023-11-02T08:48:05.680 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.680 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.680 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.680 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:48:05.681 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:05.681 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.681 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:05.681 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6814", 2023-11-02T08:48:05.681 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 135638 2023-11-02T08:48:05.681 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.682 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.682 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:05.682 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6815", 2023-11-02T08:48:05.682 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 135638 2023-11-02T08:48:05.682 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.683 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.683 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.683 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6811/135638", 2023-11-02T08:48:05.683 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6813/135638", 2023-11-02T08:48:05.683 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6817/135638", 2023-11-02T08:48:05.684 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6815/135638", 2023-11-02T08:48:05.684 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:48:05.684 INFO:tasks.workunit.client.0.smithi116.stdout: "exists", 2023-11-02T08:48:05.684 INFO:tasks.workunit.client.0.smithi116.stdout: "up" 2023-11-02T08:48:05.684 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.684 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.685 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:05.685 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_xinfo": [ 2023-11-02T08:48:05.685 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.685 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:48:05.685 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:47:12.285921+0000", 2023-11-02T08:48:05.686 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:48:05.686 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:48:05.686 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:48:05.686 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:48:05.686 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:45:06.084398+0000", 2023-11-02T08:48:05.687 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:48:05.687 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.687 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.687 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:48:05.687 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:47:12.926534+0000", 2023-11-02T08:48:05.687 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:48:05.688 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:48:05.688 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:48:05.688 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:48:05.688 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:45:39.287740+0000", 2023-11-02T08:48:05.688 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:48:05.689 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.689 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.689 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:48:05.689 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:47:13.542895+0000", 2023-11-02T08:48:05.689 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:48:05.690 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:48:05.690 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:48:05.690 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:48:05.690 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:46:13.065366+0000", 2023-11-02T08:48:05.691 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:48:05.691 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.691 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:05.691 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap": [], 2023-11-02T08:48:05.691 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap_items": [], 2023-11-02T08:48:05.692 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap_primaries": [], 2023-11-02T08:48:05.692 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_temp": [], 2023-11-02T08:48:05.692 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_temp": [], 2023-11-02T08:48:05.692 INFO:tasks.workunit.client.0.smithi116.stdout: "blocklist": {}, 2023-11-02T08:48:05.692 INFO:tasks.workunit.client.0.smithi116.stdout: "range_blocklist": {}, 2023-11-02T08:48:05.693 INFO:tasks.workunit.client.0.smithi116.stdout: "erasure_code_profiles": { 2023-11-02T08:48:05.693 INFO:tasks.workunit.client.0.smithi116.stdout: "default": { 2023-11-02T08:48:05.693 INFO:tasks.workunit.client.0.smithi116.stdout: "k": "2", 2023-11-02T08:48:05.693 INFO:tasks.workunit.client.0.smithi116.stdout: "m": "2", 2023-11-02T08:48:05.693 INFO:tasks.workunit.client.0.smithi116.stdout: "plugin": "jerasure", 2023-11-02T08:48:05.693 INFO:tasks.workunit.client.0.smithi116.stdout: "technique": "reed_sol_van" 2023-11-02T08:48:05.694 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.694 INFO:tasks.workunit.client.0.smithi116.stdout: "myprofile": { 2023-11-02T08:48:05.694 INFO:tasks.workunit.client.0.smithi116.stdout: "crush-device-class": "", 2023-11-02T08:48:05.694 INFO:tasks.workunit.client.0.smithi116.stdout: "crush-failure-domain": "osd", 2023-11-02T08:48:05.694 INFO:tasks.workunit.client.0.smithi116.stdout: "crush-root": "default", 2023-11-02T08:48:05.695 INFO:tasks.workunit.client.0.smithi116.stdout: "jerasure-per-chunk-alignment": "false", 2023-11-02T08:48:05.695 INFO:tasks.workunit.client.0.smithi116.stdout: "k": "2", 2023-11-02T08:48:05.695 INFO:tasks.workunit.client.0.smithi116.stdout: "m": "1", 2023-11-02T08:48:05.695 INFO:tasks.workunit.client.0.smithi116.stdout: "plugin": "jerasure", 2023-11-02T08:48:05.695 INFO:tasks.workunit.client.0.smithi116.stdout: "technique": "reed_sol_van", 2023-11-02T08:48:05.695 INFO:tasks.workunit.client.0.smithi116.stdout: "w": "8" 2023-11-02T08:48:05.696 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.696 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.696 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_snaps_queue": [], 2023-11-02T08:48:05.696 INFO:tasks.workunit.client.0.smithi116.stdout: "new_removed_snaps": [], 2023-11-02T08:48:05.696 INFO:tasks.workunit.client.0.smithi116.stdout: "new_purged_snaps": [], 2023-11-02T08:48:05.697 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_node_flags": {}, 2023-11-02T08:48:05.697 INFO:tasks.workunit.client.0.smithi116.stdout: "device_class_flags": {}, 2023-11-02T08:48:05.697 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode": { 2023-11-02T08:48:05.697 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode_enabled": false, 2023-11-02T08:48:05.697 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_bucket_count": 0, 2023-11-02T08:48:05.698 INFO:tasks.workunit.client.0.smithi116.stdout: "degraded_stretch_mode": 0, 2023-11-02T08:48:05.698 INFO:tasks.workunit.client.0.smithi116.stdout: "recovering_stretch_mode": 0, 2023-11-02T08:48:05.698 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode_bucket": 0 2023-11-02T08:48:05.698 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.698 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.698 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_metadata": [ 2023-11-02T08:48:05.699 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.699 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:48:05.699 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:48:05.699 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6804/135069,v1:127.0.0.1:6805/135069]", 2023-11-02T08:48:05.699 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:48:05.700 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:48:05.700 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:48:05.700 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:48:05.700 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:48:05.700 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:48:05.700 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:48:05.701 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:48:05.701 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:48:05.701 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:48:05.701 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2023-11-02T08:48:05.701 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:48:05.702 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:48:05.702 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:48:05.702 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:48:05.702 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:48:05.702 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:48:05.703 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:48:05.703 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:48:05.703 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:48:05.703 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:48:05.703 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:44:40.798381Z", 2023-11-02T08:48:05.703 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:48:05.704 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:48:05.704 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:48:05.704 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:48:05.704 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:48:05.704 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:48:05.705 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:48:05.705 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6802/135069,v1:127.0.0.1:6803/135069]", 2023-11-02T08:48:05.705 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:48:05.705 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/135069,v1:127.0.0.1:6809/135069]", 2023-11-02T08:48:05.705 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/135069,v1:127.0.0.1:6807/135069]", 2023-11-02T08:48:05.706 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:48:05.706 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:48:05.706 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:48:05.706 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:48:05.706 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:48:05.706 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:48:05.707 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:48:05.707 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:48:05.707 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:48:05.707 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/0", 2023-11-02T08:48:05.707 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:48:05.708 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:48:05.708 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:48:05.708 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.708 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.708 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:48:05.709 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:48:05.709 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6812/128752,v1:127.0.0.1:6813/128752]", 2023-11-02T08:48:05.709 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:48:05.709 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:48:05.709 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:48:05.709 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:48:05.710 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:48:05.710 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:48:05.710 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:48:05.710 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:48:05.710 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:48:05.711 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:48:05.711 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2023-11-02T08:48:05.711 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:48:05.711 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:48:05.711 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:48:05.712 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:48:05.712 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:48:05.712 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:48:05.712 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:48:05.713 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:48:05.713 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:48:05.713 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:48:05.713 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:45:13.085521Z", 2023-11-02T08:48:05.713 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:48:05.713 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:48:05.714 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:48:05.714 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:48:05.714 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:48:05.714 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:48:05.714 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:48:05.715 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6810/128752,v1:127.0.0.1:6811/128752]", 2023-11-02T08:48:05.715 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:48:05.715 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/128752,v1:127.0.0.1:6817/128752]", 2023-11-02T08:48:05.715 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/128752,v1:127.0.0.1:6815/128752]", 2023-11-02T08:48:05.715 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:48:05.716 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:48:05.716 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:48:05.716 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:48:05.716 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:48:05.716 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:48:05.716 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:48:05.717 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:48:05.717 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:48:05.717 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/1", 2023-11-02T08:48:05.717 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:48:05.717 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:48:05.718 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:48:05.718 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.718 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.718 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:48:05.719 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:48:05.719 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6812/135638,v1:127.0.0.1:6813/135638]", 2023-11-02T08:48:05.719 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:48:05.719 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:48:05.719 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:48:05.720 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:48:05.720 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:48:05.720 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:48:05.720 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:48:05.720 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:48:05.721 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:48:05.721 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:48:05.721 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2023-11-02T08:48:05.721 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:48:05.721 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:48:05.722 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:48:05.722 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:48:05.722 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:48:05.722 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:48:05.722 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:48:05.723 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:48:05.723 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:48:05.723 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:48:05.723 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:45:46.537324Z", 2023-11-02T08:48:05.723 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:48:05.723 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:48:05.724 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:48:05.724 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:48:05.724 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:48:05.724 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:48:05.724 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:48:05.725 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6810/135638,v1:127.0.0.1:6811/135638]", 2023-11-02T08:48:05.725 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:48:05.725 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/135638,v1:127.0.0.1:6817/135638]", 2023-11-02T08:48:05.725 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/135638,v1:127.0.0.1:6815/135638]", 2023-11-02T08:48:05.725 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:48:05.726 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:48:05.726 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:48:05.726 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:48:05.726 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:48:05.726 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:48:05.726 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:48:05.727 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:48:05.727 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:48:05.727 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/2", 2023-11-02T08:48:05.727 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:48:05.727 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:48:05.728 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:48:05.728 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.728 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:05.728 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_clean_epochs": { 2023-11-02T08:48:05.728 INFO:tasks.workunit.client.0.smithi116.stdout: "min_last_epoch_clean": 0, 2023-11-02T08:48:05.729 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_clean": { 2023-11-02T08:48:05.729 INFO:tasks.workunit.client.0.smithi116.stdout: "per_pool": [] 2023-11-02T08:48:05.729 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.729 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_epochs": [ 2023-11-02T08:48:05.729 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.729 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:48:05.730 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 37 2023-11-02T08:48:05.730 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.730 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.730 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:48:05.730 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 10 2023-11-02T08:48:05.731 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.731 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.731 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:48:05.731 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 40 2023-11-02T08:48:05.731 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.732 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.732 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.732 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_first_committed": 1, 2023-11-02T08:48:05.732 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_last_committed": 43, 2023-11-02T08:48:05.732 INFO:tasks.workunit.client.0.smithi116.stdout: "crushmap": { 2023-11-02T08:48:05.732 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": [ 2023-11-02T08:48:05.733 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.733 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:48:05.733 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.0", 2023-11-02T08:48:05.733 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:48:05.733 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.734 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.734 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:48:05.734 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.1", 2023-11-02T08:48:05.734 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:48:05.734 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.735 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.735 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:48:05.735 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.2", 2023-11-02T08:48:05.735 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:48:05.735 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.736 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:05.736 INFO:tasks.workunit.client.0.smithi116.stdout: "types": [ 2023-11-02T08:48:05.736 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.736 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 0, 2023-11-02T08:48:05.736 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd" 2023-11-02T08:48:05.737 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.737 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.737 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:48:05.737 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "host" 2023-11-02T08:48:05.737 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.738 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.738 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 2, 2023-11-02T08:48:05.738 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "chassis" 2023-11-02T08:48:05.738 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.738 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.739 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 3, 2023-11-02T08:48:05.739 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "rack" 2023-11-02T08:48:05.739 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.739 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.739 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 4, 2023-11-02T08:48:05.740 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "row" 2023-11-02T08:48:05.740 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.740 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.740 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 5, 2023-11-02T08:48:05.740 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "pdu" 2023-11-02T08:48:05.741 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.741 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.741 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 6, 2023-11-02T08:48:05.741 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "pod" 2023-11-02T08:48:05.741 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.741 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.742 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 7, 2023-11-02T08:48:05.742 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "room" 2023-11-02T08:48:05.742 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.742 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.742 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 8, 2023-11-02T08:48:05.743 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "datacenter" 2023-11-02T08:48:05.743 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.743 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.743 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 9, 2023-11-02T08:48:05.743 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "zone" 2023-11-02T08:48:05.743 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.744 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.744 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 10, 2023-11-02T08:48:05.744 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "region" 2023-11-02T08:48:05.744 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.744 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.745 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:48:05.745 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "root" 2023-11-02T08:48:05.745 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.745 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:05.745 INFO:tasks.workunit.client.0.smithi116.stdout: "buckets": [ 2023-11-02T08:48:05.746 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.746 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -1, 2023-11-02T08:48:05.746 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "default", 2023-11-02T08:48:05.746 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:48:05.746 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "root", 2023-11-02T08:48:05.746 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:48:05.747 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:48:05.747 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:48:05.747 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:48:05.747 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.747 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -3, 2023-11-02T08:48:05.748 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:48:05.748 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:48:05.748 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.748 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.748 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.748 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.749 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -2, 2023-11-02T08:48:05.749 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "default~hdd", 2023-11-02T08:48:05.749 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:48:05.749 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "root", 2023-11-02T08:48:05.749 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:48:05.750 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:48:05.750 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:48:05.750 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:48:05.750 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.750 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -4, 2023-11-02T08:48:05.751 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:48:05.751 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:48:05.751 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.751 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.751 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.751 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.752 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -3, 2023-11-02T08:48:05.752 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "smithi116", 2023-11-02T08:48:05.752 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:48:05.752 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "host", 2023-11-02T08:48:05.752 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:48:05.753 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:48:05.753 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:48:05.753 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:48:05.753 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.753 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:48:05.754 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:48:05.754 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:48:05.754 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.754 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.754 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:48:05.754 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:48:05.755 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 1 2023-11-02T08:48:05.755 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.755 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.755 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:48:05.755 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:48:05.756 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 2 2023-11-02T08:48:05.756 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.756 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.756 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.756 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.757 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -4, 2023-11-02T08:48:05.757 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "smithi116~hdd", 2023-11-02T08:48:05.757 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:48:05.757 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "host", 2023-11-02T08:48:05.757 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:48:05.758 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:48:05.758 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:48:05.758 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:48:05.758 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.758 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:48:05.759 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:48:05.759 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:48:05.759 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.759 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.759 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:48:05.760 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:48:05.760 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 1 2023-11-02T08:48:05.760 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.760 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.760 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:48:05.761 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:48:05.761 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 2 2023-11-02T08:48:05.761 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.761 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.761 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.762 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:05.762 INFO:tasks.workunit.client.0.smithi116.stdout: "rules": [ 2023-11-02T08:48:05.762 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.762 INFO:tasks.workunit.client.0.smithi116.stdout: "rule_id": 0, 2023-11-02T08:48:05.762 INFO:tasks.workunit.client.0.smithi116.stdout: "rule_name": "replicated_rule", 2023-11-02T08:48:05.763 INFO:tasks.workunit.client.0.smithi116.stdout: "type": 1, 2023-11-02T08:48:05.763 INFO:tasks.workunit.client.0.smithi116.stdout: "steps": [ 2023-11-02T08:48:05.763 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.763 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "take", 2023-11-02T08:48:05.763 INFO:tasks.workunit.client.0.smithi116.stdout: "item": -1, 2023-11-02T08:48:05.764 INFO:tasks.workunit.client.0.smithi116.stdout: "item_name": "default" 2023-11-02T08:48:05.764 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.764 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.764 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "choose_firstn", 2023-11-02T08:48:05.764 INFO:tasks.workunit.client.0.smithi116.stdout: "num": 0, 2023-11-02T08:48:05.764 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "osd" 2023-11-02T08:48:05.765 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.765 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.765 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "emit" 2023-11-02T08:48:05.765 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.765 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.766 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.766 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.766 INFO:tasks.workunit.client.0.smithi116.stdout: "rule_id": 1, 2023-11-02T08:48:05.766 INFO:tasks.workunit.client.0.smithi116.stdout: "rule_name": "test", 2023-11-02T08:48:05.766 INFO:tasks.workunit.client.0.smithi116.stdout: "type": 3, 2023-11-02T08:48:05.767 INFO:tasks.workunit.client.0.smithi116.stdout: "steps": [ 2023-11-02T08:48:05.767 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.767 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "set_chooseleaf_tries", 2023-11-02T08:48:05.767 INFO:tasks.workunit.client.0.smithi116.stdout: "num": 5 2023-11-02T08:48:05.767 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.767 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.768 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "set_choose_tries", 2023-11-02T08:48:05.768 INFO:tasks.workunit.client.0.smithi116.stdout: "num": 100 2023-11-02T08:48:05.768 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.768 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.768 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "take", 2023-11-02T08:48:05.769 INFO:tasks.workunit.client.0.smithi116.stdout: "item": -1, 2023-11-02T08:48:05.769 INFO:tasks.workunit.client.0.smithi116.stdout: "item_name": "default" 2023-11-02T08:48:05.769 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.769 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.769 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "choose_indep", 2023-11-02T08:48:05.769 INFO:tasks.workunit.client.0.smithi116.stdout: "num": 0, 2023-11-02T08:48:05.770 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "osd" 2023-11-02T08:48:05.770 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.770 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.770 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "emit" 2023-11-02T08:48:05.770 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.771 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:05.771 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.771 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:05.771 INFO:tasks.workunit.client.0.smithi116.stdout: "tunables": { 2023-11-02T08:48:05.771 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_local_tries": 0, 2023-11-02T08:48:05.772 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_local_fallback_tries": 0, 2023-11-02T08:48:05.772 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_total_tries": 50, 2023-11-02T08:48:05.772 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_descend_once": 1, 2023-11-02T08:48:05.772 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_vary_r": 1, 2023-11-02T08:48:05.772 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_stable": 1, 2023-11-02T08:48:05.772 INFO:tasks.workunit.client.0.smithi116.stdout: "straw_calc_version": 1, 2023-11-02T08:48:05.773 INFO:tasks.workunit.client.0.smithi116.stdout: "allowed_bucket_algs": 54, 2023-11-02T08:48:05.773 INFO:tasks.workunit.client.0.smithi116.stdout: "profile": "jewel", 2023-11-02T08:48:05.773 INFO:tasks.workunit.client.0.smithi116.stdout: "optimal_tunables": 1, 2023-11-02T08:48:05.773 INFO:tasks.workunit.client.0.smithi116.stdout: "legacy_tunables": 0, 2023-11-02T08:48:05.774 INFO:tasks.workunit.client.0.smithi116.stderr:report 1086196974 2023-11-02T08:48:05.774 INFO:tasks.workunit.client.0.smithi116.stdout: "minimum_required_version": "jewel", 2023-11-02T08:48:05.774 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables": 1, 2023-11-02T08:48:05.775 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables2": 1, 2023-11-02T08:48:05.775 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v2_rules": 1, 2023-11-02T08:48:05.775 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables3": 1, 2023-11-02T08:48:05.775 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v3_rules": 0, 2023-11-02T08:48:05.775 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v4_buckets": 1, 2023-11-02T08:48:05.776 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables5": 1, 2023-11-02T08:48:05.776 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v5_rules": 0 2023-11-02T08:48:05.776 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.777 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_args": {} 2023-11-02T08:48:05.777 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.778 INFO:tasks.workunit.client.0.smithi116.stdout: "fsmap": { 2023-11-02T08:48:05.778 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 1, 2023-11-02T08:48:05.778 INFO:tasks.workunit.client.0.smithi116.stdout: "default_fscid": -1, 2023-11-02T08:48:05.778 INFO:tasks.workunit.client.0.smithi116.stdout: "compat": { 2023-11-02T08:48:05.778 INFO:tasks.workunit.client.0.smithi116.stdout: "compat": {}, 2023-11-02T08:48:05.779 INFO:tasks.workunit.client.0.smithi116.stdout: "ro_compat": {}, 2023-11-02T08:48:05.779 INFO:tasks.workunit.client.0.smithi116.stdout: "incompat": { 2023-11-02T08:48:05.779 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_1": "base v0.20", 2023-11-02T08:48:05.779 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_2": "client writeable ranges", 2023-11-02T08:48:05.779 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_3": "default file layouts on dirs", 2023-11-02T08:48:05.779 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_4": "dir inode in separate object", 2023-11-02T08:48:05.780 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_5": "mds uses versioned encoding", 2023-11-02T08:48:05.780 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_6": "dirfrag is stored in omap", 2023-11-02T08:48:05.780 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_8": "no anchor table", 2023-11-02T08:48:05.780 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_9": "file layout v2", 2023-11-02T08:48:05.780 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_10": "snaprealm v2" 2023-11-02T08:48:05.781 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.781 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.781 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_flags": { 2023-11-02T08:48:05.781 INFO:tasks.workunit.client.0.smithi116.stdout: "enable_multiple": true, 2023-11-02T08:48:05.781 INFO:tasks.workunit.client.0.smithi116.stdout: "ever_enabled_multiple": true 2023-11-02T08:48:05.782 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.782 INFO:tasks.workunit.client.0.smithi116.stdout: "standbys": [], 2023-11-02T08:48:05.782 INFO:tasks.workunit.client.0.smithi116.stdout: "filesystems": [] 2023-11-02T08:48:05.782 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.782 INFO:tasks.workunit.client.0.smithi116.stdout: "mdsmap_first_committed": 1, 2023-11-02T08:48:05.782 INFO:tasks.workunit.client.0.smithi116.stdout: "mdsmap_last_committed": 1, 2023-11-02T08:48:05.783 INFO:tasks.workunit.client.0.smithi116.stdout: "auth": { 2023-11-02T08:48:05.783 INFO:tasks.workunit.client.0.smithi116.stdout: "first_committed": 1, 2023-11-02T08:48:05.783 INFO:tasks.workunit.client.0.smithi116.stdout: "last_committed": 5, 2023-11-02T08:48:05.783 INFO:tasks.workunit.client.0.smithi116.stdout: "num_secrets": 3 2023-11-02T08:48:05.783 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.784 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg": 1, 2023-11-02T08:48:05.784 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_active": 1, 2023-11-02T08:48:05.784 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_unknown": 0, 2023-11-02T08:48:05.784 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osd": 3, 2023-11-02T08:48:05.784 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_sum": { 2023-11-02T08:48:05.784 INFO:tasks.workunit.client.0.smithi116.stdout: "stat_sum": { 2023-11-02T08:48:05.785 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes": 48430, 2023-11-02T08:48:05.785 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects": 101, 2023-11-02T08:48:05.785 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_clones": 1, 2023-11-02T08:48:05.785 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_copies": 303, 2023-11-02T08:48:05.785 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing_on_primary": 0, 2023-11-02T08:48:05.786 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing": 0, 2023-11-02T08:48:05.786 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_degraded": 101, 2023-11-02T08:48:05.786 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_misplaced": 0, 2023-11-02T08:48:05.786 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_unfound": 0, 2023-11-02T08:48:05.786 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_dirty": 101, 2023-11-02T08:48:05.787 INFO:tasks.workunit.client.0.smithi116.stdout: "num_whiteouts": 0, 2023-11-02T08:48:05.787 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read": 0, 2023-11-02T08:48:05.787 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read_kb": 0, 2023-11-02T08:48:05.787 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write": 101, 2023-11-02T08:48:05.787 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write_kb": 101, 2023-11-02T08:48:05.787 INFO:tasks.workunit.client.0.smithi116.stdout: "num_scrub_errors": 0, 2023-11-02T08:48:05.788 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shallow_scrub_errors": 0, 2023-11-02T08:48:05.788 INFO:tasks.workunit.client.0.smithi116.stdout: "num_deep_scrub_errors": 0, 2023-11-02T08:48:05.788 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_recovered": 0, 2023-11-02T08:48:05.788 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_recovered": 0, 2023-11-02T08:48:05.788 INFO:tasks.workunit.client.0.smithi116.stdout: "num_keys_recovered": 0, 2023-11-02T08:48:05.789 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_omap": 0, 2023-11-02T08:48:05.789 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_hit_set_archive": 0, 2023-11-02T08:48:05.789 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_hit_set_archive": 0, 2023-11-02T08:48:05.789 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush": 0, 2023-11-02T08:48:05.789 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_kb": 0, 2023-11-02T08:48:05.790 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict": 0, 2023-11-02T08:48:05.790 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_kb": 0, 2023-11-02T08:48:05.790 INFO:tasks.workunit.client.0.smithi116.stdout: "num_promote": 0, 2023-11-02T08:48:05.790 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_high": 0, 2023-11-02T08:48:05.790 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_low": 0, 2023-11-02T08:48:05.790 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_some": 0, 2023-11-02T08:48:05.791 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_full": 0, 2023-11-02T08:48:05.791 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_pinned": 0, 2023-11-02T08:48:05.791 INFO:tasks.workunit.client.0.smithi116.stdout: "num_legacy_snapsets": 0, 2023-11-02T08:48:05.791 INFO:tasks.workunit.client.0.smithi116.stdout: "num_large_omap_objects": 0, 2023-11-02T08:48:05.791 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_manifest": 0, 2023-11-02T08:48:05.792 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_bytes": 0, 2023-11-02T08:48:05.792 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_keys": 0, 2023-11-02T08:48:05.792 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_repaired": 0 2023-11-02T08:48:05.792 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.792 INFO:tasks.workunit.client.0.smithi116.stdout: "store_stats": { 2023-11-02T08:48:05.793 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 0, 2023-11-02T08:48:05.793 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 0, 2023-11-02T08:48:05.793 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:48:05.793 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 0, 2023-11-02T08:48:05.793 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 0, 2023-11-02T08:48:05.793 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:48:05.794 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:48:05.794 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:48:05.794 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 0, 2023-11-02T08:48:05.794 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 0 2023-11-02T08:48:05.794 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.795 INFO:tasks.workunit.client.0.smithi116.stdout: "log_size": 18, 2023-11-02T08:48:05.795 INFO:tasks.workunit.client.0.smithi116.stdout: "ondisk_log_size": 18, 2023-11-02T08:48:05.795 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 3, 2023-11-02T08:48:05.795 INFO:tasks.workunit.client.0.smithi116.stdout: "acting": 3, 2023-11-02T08:48:05.795 INFO:tasks.workunit.client.0.smithi116.stdout: "num_store_stats": 0 2023-11-02T08:48:05.796 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.796 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_sum": { 2023-11-02T08:48:05.796 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 0, 2023-11-02T08:48:05.796 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 0, 2023-11-02T08:48:05.796 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pgs": 3, 2023-11-02T08:48:05.796 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osds": 3, 2023-11-02T08:48:05.797 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_osds": 3, 2023-11-02T08:48:05.797 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_omap_osds": 3, 2023-11-02T08:48:05.797 INFO:tasks.workunit.client.0.smithi116.stdout: "kb": 314572800, 2023-11-02T08:48:05.797 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used": 83348, 2023-11-02T08:48:05.797 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_data": 2164, 2023-11-02T08:48:05.798 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_omap": 2, 2023-11-02T08:48:05.798 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_meta": 81149, 2023-11-02T08:48:05.798 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_avail": 314489452, 2023-11-02T08:48:05.798 INFO:tasks.workunit.client.0.smithi116.stdout: "statfs": { 2023-11-02T08:48:05.798 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 322122547200, 2023-11-02T08:48:05.799 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 322037198848, 2023-11-02T08:48:05.799 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:48:05.799 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 2215936, 2023-11-02T08:48:05.799 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 1546649, 2023-11-02T08:48:05.799 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:48:05.799 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:48:05.800 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:48:05.800 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 2366, 2023-11-02T08:48:05.800 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 83097282 2023-11-02T08:48:05.800 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.800 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_peers": [], 2023-11-02T08:48:05.801 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_trim_queue_len": 0, 2023-11-02T08:48:05.801 INFO:tasks.workunit.client.0.smithi116.stdout: "num_snap_trimming": 0, 2023-11-02T08:48:05.801 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shards_repaired": 0, 2023-11-02T08:48:05.801 INFO:tasks.workunit.client.0.smithi116.stdout: "op_queue_age_hist": { 2023-11-02T08:48:05.801 INFO:tasks.workunit.client.0.smithi116.stdout: "histogram": [], 2023-11-02T08:48:05.802 INFO:tasks.workunit.client.0.smithi116.stdout: "upper_bound": 1 2023-11-02T08:48:05.802 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.802 INFO:tasks.workunit.client.0.smithi116.stdout: "perf_stat": { 2023-11-02T08:48:05.802 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ms": 0, 2023-11-02T08:48:05.802 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ms": 0, 2023-11-02T08:48:05.802 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ns": 0, 2023-11-02T08:48:05.803 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ns": 0 2023-11-02T08:48:05.803 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.803 INFO:tasks.workunit.client.0.smithi116.stdout: "alerts": [], 2023-11-02T08:48:05.803 INFO:tasks.workunit.client.0.smithi116.stdout: "network_ping_times": [] 2023-11-02T08:48:05.803 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.804 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_sum_by_class": { 2023-11-02T08:48:05.804 INFO:tasks.workunit.client.0.smithi116.stdout: "hdd": { 2023-11-02T08:48:05.804 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 0, 2023-11-02T08:48:05.804 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 0, 2023-11-02T08:48:05.804 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pgs": 3, 2023-11-02T08:48:05.805 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osds": 3, 2023-11-02T08:48:05.805 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_osds": 3, 2023-11-02T08:48:05.805 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_omap_osds": 3, 2023-11-02T08:48:05.805 INFO:tasks.workunit.client.0.smithi116.stdout: "kb": 314572800, 2023-11-02T08:48:05.805 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used": 83348, 2023-11-02T08:48:05.805 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_data": 2164, 2023-11-02T08:48:05.806 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_omap": 2, 2023-11-02T08:48:05.806 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_meta": 81149, 2023-11-02T08:48:05.806 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_avail": 314489452, 2023-11-02T08:48:05.806 INFO:tasks.workunit.client.0.smithi116.stdout: "statfs": { 2023-11-02T08:48:05.806 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 322122547200, 2023-11-02T08:48:05.807 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 322037198848, 2023-11-02T08:48:05.807 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:48:05.807 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 2215936, 2023-11-02T08:48:05.807 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 1546649, 2023-11-02T08:48:05.807 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:48:05.807 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:48:05.808 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:48:05.808 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 2366, 2023-11-02T08:48:05.808 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 83097282 2023-11-02T08:48:05.808 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.808 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_peers": [], 2023-11-02T08:48:05.809 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_trim_queue_len": 0, 2023-11-02T08:48:05.809 INFO:tasks.workunit.client.0.smithi116.stdout: "num_snap_trimming": 0, 2023-11-02T08:48:05.809 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shards_repaired": 0, 2023-11-02T08:48:05.809 INFO:tasks.workunit.client.0.smithi116.stdout: "op_queue_age_hist": { 2023-11-02T08:48:05.809 INFO:tasks.workunit.client.0.smithi116.stdout: "histogram": [], 2023-11-02T08:48:05.809 INFO:tasks.workunit.client.0.smithi116.stdout: "upper_bound": 1 2023-11-02T08:48:05.810 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.810 INFO:tasks.workunit.client.0.smithi116.stdout: "perf_stat": { 2023-11-02T08:48:05.810 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ms": 0, 2023-11-02T08:48:05.810 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ms": 0, 2023-11-02T08:48:05.810 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ns": 0, 2023-11-02T08:48:05.811 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ns": 0 2023-11-02T08:48:05.811 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.811 INFO:tasks.workunit.client.0.smithi116.stdout: "alerts": [], 2023-11-02T08:48:05.811 INFO:tasks.workunit.client.0.smithi116.stdout: "network_ping_times": [] 2023-11-02T08:48:05.811 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.812 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.812 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_stats": [ 2023-11-02T08:48:05.812 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.812 INFO:tasks.workunit.client.0.smithi116.stdout: "poolid": 1, 2023-11-02T08:48:05.812 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg": 1, 2023-11-02T08:48:05.812 INFO:tasks.workunit.client.0.smithi116.stdout: "stat_sum": { 2023-11-02T08:48:05.813 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes": 48430, 2023-11-02T08:48:05.813 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects": 101, 2023-11-02T08:48:05.813 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_clones": 1, 2023-11-02T08:48:05.813 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_copies": 303, 2023-11-02T08:48:05.813 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing_on_primary": 0, 2023-11-02T08:48:05.814 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing": 0, 2023-11-02T08:48:05.814 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_degraded": 101, 2023-11-02T08:48:05.814 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_misplaced": 0, 2023-11-02T08:48:05.814 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_unfound": 0, 2023-11-02T08:48:05.814 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_dirty": 101, 2023-11-02T08:48:05.815 INFO:tasks.workunit.client.0.smithi116.stdout: "num_whiteouts": 0, 2023-11-02T08:48:05.815 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read": 0, 2023-11-02T08:48:05.815 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read_kb": 0, 2023-11-02T08:48:05.815 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write": 101, 2023-11-02T08:48:05.815 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write_kb": 101, 2023-11-02T08:48:05.815 INFO:tasks.workunit.client.0.smithi116.stdout: "num_scrub_errors": 0, 2023-11-02T08:48:05.816 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shallow_scrub_errors": 0, 2023-11-02T08:48:05.816 INFO:tasks.workunit.client.0.smithi116.stdout: "num_deep_scrub_errors": 0, 2023-11-02T08:48:05.816 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_recovered": 0, 2023-11-02T08:48:05.816 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_recovered": 0, 2023-11-02T08:48:05.816 INFO:tasks.workunit.client.0.smithi116.stdout: "num_keys_recovered": 0, 2023-11-02T08:48:05.817 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_omap": 0, 2023-11-02T08:48:05.817 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_hit_set_archive": 0, 2023-11-02T08:48:05.817 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_hit_set_archive": 0, 2023-11-02T08:48:05.817 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush": 0, 2023-11-02T08:48:05.817 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_kb": 0, 2023-11-02T08:48:05.817 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict": 0, 2023-11-02T08:48:05.818 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_kb": 0, 2023-11-02T08:48:05.818 INFO:tasks.workunit.client.0.smithi116.stdout: "num_promote": 0, 2023-11-02T08:48:05.818 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_high": 0, 2023-11-02T08:48:05.818 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_low": 0, 2023-11-02T08:48:05.818 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_some": 0, 2023-11-02T08:48:05.819 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_full": 0, 2023-11-02T08:48:05.819 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_pinned": 0, 2023-11-02T08:48:05.819 INFO:tasks.workunit.client.0.smithi116.stdout: "num_legacy_snapsets": 0, 2023-11-02T08:48:05.819 INFO:tasks.workunit.client.0.smithi116.stdout: "num_large_omap_objects": 0, 2023-11-02T08:48:05.819 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_manifest": 0, 2023-11-02T08:48:05.820 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_bytes": 0, 2023-11-02T08:48:05.820 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_keys": 0, 2023-11-02T08:48:05.820 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_repaired": 0 2023-11-02T08:48:05.820 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.820 INFO:tasks.workunit.client.0.smithi116.stdout: "store_stats": { 2023-11-02T08:48:05.820 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 0, 2023-11-02T08:48:05.821 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 0, 2023-11-02T08:48:05.821 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:48:05.821 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 1241088, 2023-11-02T08:48:05.821 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 1241088, 2023-11-02T08:48:05.821 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:48:05.822 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:48:05.822 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:48:05.822 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 0, 2023-11-02T08:48:05.822 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 0 2023-11-02T08:48:05.822 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.823 INFO:tasks.workunit.client.0.smithi116.stdout: "log_size": 18, 2023-11-02T08:48:05.823 INFO:tasks.workunit.client.0.smithi116.stdout: "ondisk_log_size": 18, 2023-11-02T08:48:05.823 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 3, 2023-11-02T08:48:05.823 INFO:tasks.workunit.client.0.smithi116.stdout: "acting": 3, 2023-11-02T08:48:05.823 INFO:tasks.workunit.client.0.smithi116.stdout: "num_store_stats": 3 2023-11-02T08:48:05.824 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.824 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:05.824 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_stats": [ 2023-11-02T08:48:05.824 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.824 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:48:05.824 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 158913789963 2023-11-02T08:48:05.825 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.825 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.825 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:48:05.825 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 42949672988 2023-11-02T08:48:05.825 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.826 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.826 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:48:05.826 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 167503724555 2023-11-02T08:48:05.826 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.826 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:05.827 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_by_state": [ 2023-11-02T08:48:05.827 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.827 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "active+undersized+degraded", 2023-11-02T08:48:05.827 INFO:tasks.workunit.client.0.smithi116.stdout: "num": 1 2023-11-02T08:48:05.827 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.827 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:05.828 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_by_osd": [ 2023-11-02T08:48:05.828 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.828 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2147483647, 2023-11-02T08:48:05.828 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 0, 2023-11-02T08:48:05.828 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 1, 2023-11-02T08:48:05.829 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:48:05.829 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.829 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.829 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:48:05.829 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 0, 2023-11-02T08:48:05.830 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 1, 2023-11-02T08:48:05.830 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:48:05.831 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.831 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.831 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:48:05.831 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 1, 2023-11-02T08:48:05.831 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 1, 2023-11-02T08:48:05.832 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:48:05.832 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.832 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.832 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:48:05.832 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 0, 2023-11-02T08:48:05.833 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 0, 2023-11-02T08:48:05.833 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:48:05.833 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.833 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:05.833 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snaps": [ 2023-11-02T08:48:05.833 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:05.834 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": 1, 2023-11-02T08:48:05.834 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snaps": {} 2023-11-02T08:48:05.834 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.834 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:05.834 INFO:tasks.workunit.client.0.smithi116.stdout: "servicemap": { 2023-11-02T08:48:05.835 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 5, 2023-11-02T08:48:05.835 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:47:15.584380+0000", 2023-11-02T08:48:05.835 INFO:tasks.workunit.client.0.smithi116.stdout: "services": {} 2023-11-02T08:48:05.835 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:05.835 INFO:tasks.workunit.client.0.smithi116.stdout: "mgrstat_first_committed": 1, 2023-11-02T08:48:05.836 INFO:tasks.workunit.client.0.smithi116.stdout: "mgrstat_last_committed": 94, 2023-11-02T08:48:05.836 INFO:tasks.workunit.client.0.smithi116.stdout: "logm_first_committed": 1, 2023-11-02T08:48:05.836 INFO:tasks.workunit.client.0.smithi116.stdout: "logm_last_committed": 388, 2023-11-02T08:48:05.836 INFO:tasks.workunit.client.0.smithi116.stdout: "paxos": { 2023-11-02T08:48:05.836 INFO:tasks.workunit.client.0.smithi116.stdout: "first_committed": 1, 2023-11-02T08:48:05.836 INFO:tasks.workunit.client.0.smithi116.stdout: "last_committed": 568, 2023-11-02T08:48:05.837 INFO:tasks.workunit.client.0.smithi116.stdout: "last_pn": 0, 2023-11-02T08:48:05.837 INFO:tasks.workunit.client.0.smithi116.stdout: "accepted_pn": 0 2023-11-02T08:48:05.837 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:05.837 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T08:48:05.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2023-11-02T08:48:05.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2023-11-02T08:48:05.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2023-11-02T08:48:05.840 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:48:05.840 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:48:05.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:48:05.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:48:05.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:48:05.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:48:05.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2023-11-02T08:48:05.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2023-11-02T08:48:05.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2023-11-02T08:48:05.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2023-11-02T08:48:05.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2023-11-02T08:48:05.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2023-11-02T08:48:05.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2023-11-02T08:48:06.730 INFO:tasks.workunit.client.0.smithi116.stderr:PG '1.0' not found 2023-11-02T08:48:07.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2023-11-02T08:48:07.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2023-11-02T08:48:07.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2023-11-02T08:48:07.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:48:07.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-11-02T08:48:07.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:48:07.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2023-11-02T08:48:07.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:48:07.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:48:07.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:48:07.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:48:07.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2023-11-02T08:48:07.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2023-11-02T08:48:07.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:48:07.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:48:07.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:48:07.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:48:07.007 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:48:07.007 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:48:07.007 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:48:07.007 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:48:07.007 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:48:07.008 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:48:07.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:48:07.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:48:07.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:48:07.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:48:07.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:48:07.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:48:07.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:48:07.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:48:07.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:48:07.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:48:07.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2023-11-02T08:48:07.015 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T08:48:07.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-11-02T08:48:07.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:48:07.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2023-11-02T08:48:07.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-11-02T08:48:07.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:48:07.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:48:07.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:48:07.039 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:48:07.036+0000 7f710fa0c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:48:07.041 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:48:07.039+0000 7f710fa0c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:48:07.043 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:48:07.040+0000 7f710fa0c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:48:07.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-11-02T08:48:07.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:48:07.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:48:07.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:48:07.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:48:07.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:48:07.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:48:07.427 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:48:07.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:48:07.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:48:07.570 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:48:07.567+0000 7f710fa0c500 -1 Falling back to public interface 2023-11-02T08:48:07.848 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 37 up_thru 42 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/135069,v1:127.0.0.1:6803/135069] [v2:127.0.0.1:6804/135069,v1:127.0.0.1:6805/135069] exists,up 5f38bac9-3e89-4df4-9a41-799f865c97c8 2023-11-02T08:48:07.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:48:07.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:48:07.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:48:07.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2023-11-02T08:48:07.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2023-11-02T08:48:07.851 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:48:07.851 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:48:07.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:48:07.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:48:07.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:48:07.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:48:07.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2023-11-02T08:48:07.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2023-11-02T08:48:07.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2023-11-02T08:48:07.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2023-11-02T08:48:07.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2023-11-02T08:48:07.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2023-11-02T08:48:07.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2023-11-02T08:48:08.997 INFO:tasks.workunit.client.0.smithi116.stderr:PG '1.0' not found 2023-11-02T08:48:09.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2023-11-02T08:48:09.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2023-11-02T08:48:09.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2023-11-02T08:48:09.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:48:09.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2023-11-02T08:48:09.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:48:09.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2023-11-02T08:48:09.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:48:09.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:48:09.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:48:09.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:48:09.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2023-11-02T08:48:09.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2023-11-02T08:48:09.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:48:09.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:48:09.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:48:09.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:48:09.274 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:48:09.274 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:48:09.274 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:48:09.274 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:48:09.274 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:48:09.275 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:48:09.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:48:09.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:48:09.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:48:09.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:48:09.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:48:09.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:48:09.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:48:09.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:48:09.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:48:09.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:48:09.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2023-11-02T08:48:09.277 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T08:48:09.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2023-11-02T08:48:09.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:48:09.279 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2023-11-02T08:48:09.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2023-11-02T08:48:09.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:48:09.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:48:09.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:48:09.306 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:48:09.303+0000 7f9ccf40e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:48:09.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:48:09.306+0000 7f9ccf40e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:48:09.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:48:09.307+0000 7f9ccf40e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:48:09.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2023-11-02T08:48:09.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:48:09.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T08:48:09.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:48:09.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:48:09.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:48:09.666 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:48:09.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:48:09.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:48:09.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:48:09.830 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:48:09.827+0000 7f9ccf40e500 -1 Falling back to public interface 2023-11-02T08:48:10.044 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:48:10.042+0000 7f710fa0c500 -1 osd.0 43 log_to_monitors true 2023-11-02T08:48:10.197 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/135638,v1:127.0.0.1:6811/135638] [v2:127.0.0.1:6812/135638,v1:127.0.0.1:6813/135638] exists,up c3024fa8-7c9d-4674-b3d4-e8b9122765a2 2023-11-02T08:48:10.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:48:10.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:48:10.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:48:10.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2023-11-02T08:48:10.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2023-11-02T08:48:10.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2023-11-02T08:48:10.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2023-11-02T08:48:10.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2023-11-02T08:48:10.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2023-11-02T08:48:10.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2023-11-02T08:48:11.702 INFO:tasks.workunit.client.0.smithi116.stderr:PG '1.0' not found 2023-11-02T08:48:11.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2023-11-02T08:48:11.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2023-11-02T08:48:11.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2023-11-02T08:48:11.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T08:48:11.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T08:48:11.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T08:48:11.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2023-11-02T08:48:11.974 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T08:48:11.974 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T08:48:11.974 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T08:48:11.974 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T08:48:11.975 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T08:48:12.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2023-11-02T08:48:12.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T08:48:12.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T08:48:12.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T08:48:12.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:48:12.032 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:48:12.229 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:48:12.226+0000 7f9ccf40e500 -1 osd.2 43 log_to_monitors true 2023-11-02T08:48:12.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:48:12.414 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:48:12.415 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T08:48:12.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:48:12.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:48:12.415 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:48:12.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=197568495619 2023-11-02T08:48:12.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 197568495619 2023-11-02T08:48:12.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-197568495619' 2023-11-02T08:48:12.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:48:12.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:48:12.620 INFO:tasks.workunit.client.0.smithi116.stderr:Error ENXIO: problem getting command descriptions from osd.1 2023-11-02T08:48:12.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2023-11-02T08:48:12.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2023-11-02T08:48:12.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2023-11-02T08:48:12.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:48:12.623 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:48:12.709 INFO:tasks.workunit.client.0.smithi116.stderr:Error ENXIO: problem getting command descriptions from osd.2 2023-11-02T08:48:12.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2023-11-02T08:48:12.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2023-11-02T08:48:12.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2023-11-02T08:48:12.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:48:12.712 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-197568495619 2023-11-02T08:48:12.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:48:12.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:48:12.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-197568495619 2023-11-02T08:48:12.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:48:12.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=197568495619 2023-11-02T08:48:12.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 197568495619' 2023-11-02T08:48:12.715 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 197568495619 2023-11-02T08:48:12.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:48:13.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 1 -lt 197568495619 2023-11-02T08:48:13.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T08:48:14.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T08:48:14.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:48:14.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 197568495619 -lt 197568495619 2023-11-02T08:48:14.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T08:48:14.634 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:48:14.634 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:48:15.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2023-11-02T08:48:15.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:48:15.141 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:48:15.141 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:48:15.142 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:48:15.142 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:48:15.142 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:48:15.143 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:48:15.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:48:15.512 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:48:15.512 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:48:15.512 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:48:16.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:48:16.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2023-11-02T08:48:16.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2023-11-02T08:48:16.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2023-11-02T08:48:16.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:48:16.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T08:48:16.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:48:16.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:48:16.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:48:16.128 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:48:16.128 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:48:16.128 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:48:16.128 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:48:16.128 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:48:16.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:48:16.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:48:16.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:48:16.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:48:17.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:48:17.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:48:17.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:48:17.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:48:17.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:48:17.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:48:17.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:48:17.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:48:17.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:48:17.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:48:17.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:48:17.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2023-11-02T08:48:17.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:48:17.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2023-11-02T08:48:17.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:48:17.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:48:17.765 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:48:17.765 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:48:17.765 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:48:17.765 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:48:17.766 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:48:17.766 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:48:18.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:48:18.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:48:18.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:48:18.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:48:18.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:48:18.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:48:18.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:48:18.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:48:18.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:48:18.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:48:18.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:48:18.647 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:48:18.647 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:48:19.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:48:19.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:48:19.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2023-11-02T08:48:19.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:48:19.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2023-11-02T08:48:19.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:48:19.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:48:19.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:48:19.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:48:19.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:48:19.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:48:19.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:48:19.588 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:48:19.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:48:19.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:48:19.958 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:48:19.958 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:48:20.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:48:20.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:48:20.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:48:20.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:48:20.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:48:20.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:48:20.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:48:20.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:48:20.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:48:21.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:48:21.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:48:21.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2023-11-02T08:48:21.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:48:21.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2023-11-02T08:48:21.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:48:21.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:48:21.832 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:48:21.832 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:48:21.832 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:48:21.833 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:48:21.833 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:48:21.833 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:48:22.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:48:22.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:48:22.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:48:22.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:48:22.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:48:22.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:48:22.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:48:22.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:48:22.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:48:22.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:48:22.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:48:22.703 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:48:22.703 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:48:23.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:48:23.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:48:23.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2023-11-02T08:48:23.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:48:23.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2023-11-02T08:48:24.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:48:24.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:48:24.828 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:48:24.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:48:24.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:48:24.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:48:24.830 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:48:24.830 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:48:25.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:48:25.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:48:25.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:48:25.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:48:25.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:48:25.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:48:25.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:48:25.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:48:25.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:48:25.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:48:25.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:48:25.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:48:25.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:48:26.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:48:26.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:48:26.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2023-11-02T08:48:26.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:48:26.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2023-11-02T08:48:29.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:48:29.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:48:29.462 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:48:29.462 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:48:29.462 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:48:29.463 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:48:29.463 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:48:29.463 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:48:29.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:48:29.847 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:48:29.847 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:48:29.848 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:48:30.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:48:30.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:48:30.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:48:30.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:48:30.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:48:30.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:48:30.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:48:30.355 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:48:30.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:48:30.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:48:30.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:48:30.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2023-11-02T08:48:30.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:48:30.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2023-11-02T08:48:37.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:48:37.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:48:37.302 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:48:37.302 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:48:37.302 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:48:37.302 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:48:37.303 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:48:37.303 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:48:37.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:48:37.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:48:37.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:48:37.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:48:38.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:48:38.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:48:38.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:48:38.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:48:38.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:48:38.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:48:38.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:48:38.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:48:38.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:48:38.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:48:38.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:48:38.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2023-11-02T08:48:38.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T08:48:38.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2023-11-02T08:48:46.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T08:48:46.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:48:46.042 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:48:46.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:48:46.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:48:46.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:48:46.044 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:48:46.044 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:48:46.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T08:48:46.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:48:46.420 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:48:46.420 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:48:46.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2023-11-02T08:48:46.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T08:48:46.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T08:48:46.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T08:48:46.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T08:48:46.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T08:48:46.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T08:48:46.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T08:48:46.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T08:48:47.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-11-02T08:48:47.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-11-02T08:48:47.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2023-11-02T08:48:47.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2023-11-02T08:48:48.040 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T08:48:48.041 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_fingerprint": "2a860d17-8260-45c6-94e5-d9c79ce24314", 2023-11-02T08:48:48.041 INFO:tasks.workunit.client.0.smithi116.stdout: "version": "18.2.0-1159-gb5a9486b", 2023-11-02T08:48:48.041 INFO:tasks.workunit.client.0.smithi116.stdout: "commit": "b5a9486b4a6f144f83c51b7ec33111a2c86dc081", 2023-11-02T08:48:48.041 INFO:tasks.workunit.client.0.smithi116.stdout: "timestamp": "2023-11-02T08:48:48.037230+0000", 2023-11-02T08:48:48.042 INFO:tasks.workunit.client.0.smithi116.stdout: "tag": "", 2023-11-02T08:48:48.042 INFO:tasks.workunit.client.0.smithi116.stdout: "health": { 2023-11-02T08:48:48.042 INFO:tasks.workunit.client.0.smithi116.stdout: "status": "HEALTH_WARN", 2023-11-02T08:48:48.042 INFO:tasks.workunit.client.0.smithi116.stdout: "checks": { 2023-11-02T08:48:48.042 INFO:tasks.workunit.client.0.smithi116.stdout: "OSDMAP_FLAGS": { 2023-11-02T08:48:48.043 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:48:48.043 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:48:48.043 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "nodown,noout,noin flag(s) set", 2023-11-02T08:48:48.043 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 17 2023-11-02T08:48:48.043 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.044 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [], 2023-11-02T08:48:48.044 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:48:48.044 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.045 INFO:tasks.workunit.client.0.smithi116.stdout: "OSD_DOWN": { 2023-11-02T08:48:48.045 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:48:48.045 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:48:48.045 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "1 osds down", 2023-11-02T08:48:48.046 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 1 2023-11-02T08:48:48.046 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.046 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:48:48.046 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.047 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "osd.1 (root=default,host=smithi116) is down" 2023-11-02T08:48:48.047 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.047 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:48.049 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:48:48.049 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.050 INFO:tasks.workunit.client.0.smithi116.stdout: "PG_DEGRADED": { 2023-11-02T08:48:48.050 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:48:48.050 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:48:48.050 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2023-11-02T08:48:48.050 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 1 2023-11-02T08:48:48.051 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.051 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:48:48.051 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.051 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2023-11-02T08:48:48.051 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.052 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:48.052 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:48:48.052 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.052 INFO:tasks.workunit.client.0.smithi116.stdout: "POOL_APP_NOT_ENABLED": { 2023-11-02T08:48:48.052 INFO:tasks.workunit.client.0.smithi116.stdout: "severity": "HEALTH_WARN", 2023-11-02T08:48:48.053 INFO:tasks.workunit.client.0.smithi116.stdout: "summary": { 2023-11-02T08:48:48.053 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "1 pool(s) do not have an application enabled", 2023-11-02T08:48:48.053 INFO:tasks.workunit.client.0.smithi116.stdout: "count": 1 2023-11-02T08:48:48.053 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.054 INFO:tasks.workunit.client.0.smithi116.stdout: "detail": [ 2023-11-02T08:48:48.054 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.054 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "application not enabled on pool 'test'" 2023-11-02T08:48:48.054 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.054 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.055 INFO:tasks.workunit.client.0.smithi116.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2023-11-02T08:48:48.055 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.055 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:48.055 INFO:tasks.workunit.client.0.smithi116.stdout: "muted": false 2023-11-02T08:48:48.055 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.056 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.056 INFO:tasks.workunit.client.0.smithi116.stdout: "mutes": [] 2023-11-02T08:48:48.056 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.056 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap_first_committed": 1, 2023-11-02T08:48:48.056 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap_last_committed": 1, 2023-11-02T08:48:48.057 INFO:tasks.workunit.client.0.smithi116.stdout: "monmap": { 2023-11-02T08:48:48.057 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 1, 2023-11-02T08:48:48.057 INFO:tasks.workunit.client.0.smithi116.stdout: "fsid": "0c9a372f-cec7-47d9-a309-8248453aabeb", 2023-11-02T08:48:48.060 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:44:34.120499Z", 2023-11-02T08:48:48.060 INFO:tasks.workunit.client.0.smithi116.stdout: "created": "2023-11-02T08:44:34.120499Z", 2023-11-02T08:48:48.060 INFO:tasks.workunit.client.0.smithi116.stdout: "min_mon_release": 18, 2023-11-02T08:48:48.061 INFO:tasks.workunit.client.0.smithi116.stdout: "min_mon_release_name": "reef", 2023-11-02T08:48:48.061 INFO:tasks.workunit.client.0.smithi116.stdout: "election_strategy": 1, 2023-11-02T08:48:48.061 INFO:tasks.workunit.client.0.smithi116.stdout: "disallowed_leaders: ": "", 2023-11-02T08:48:48.061 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode": false, 2023-11-02T08:48:48.061 INFO:tasks.workunit.client.0.smithi116.stdout: "tiebreaker_mon": "", 2023-11-02T08:48:48.061 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_ranks: ": "", 2023-11-02T08:48:48.062 INFO:tasks.workunit.client.0.smithi116.stdout: "features": { 2023-11-02T08:48:48.062 INFO:tasks.workunit.client.0.smithi116.stdout: "persistent": [ 2023-11-02T08:48:48.063 INFO:tasks.workunit.client.0.smithi116.stdout: "kraken", 2023-11-02T08:48:48.063 INFO:tasks.workunit.client.0.smithi116.stdout: "luminous", 2023-11-02T08:48:48.063 INFO:tasks.workunit.client.0.smithi116.stdout: "mimic", 2023-11-02T08:48:48.063 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap-prune", 2023-11-02T08:48:48.063 INFO:tasks.workunit.client.0.smithi116.stdout: "nautilus", 2023-11-02T08:48:48.063 INFO:tasks.workunit.client.0.smithi116.stdout: "octopus", 2023-11-02T08:48:48.064 INFO:tasks.workunit.client.0.smithi116.stdout: "pacific", 2023-11-02T08:48:48.064 INFO:tasks.workunit.client.0.smithi116.stdout: "elector-pinging", 2023-11-02T08:48:48.064 INFO:tasks.workunit.client.0.smithi116.stdout: "quincy", 2023-11-02T08:48:48.064 INFO:tasks.workunit.client.0.smithi116.stdout: "reef" 2023-11-02T08:48:48.064 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:48.065 INFO:tasks.workunit.client.0.smithi116.stdout: "optional": [] 2023-11-02T08:48:48.065 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.065 INFO:tasks.workunit.client.0.smithi116.stdout: "mons": [ 2023-11-02T08:48:48.065 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.066 INFO:tasks.workunit.client.0.smithi116.stdout: "rank": 0, 2023-11-02T08:48:48.066 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "a", 2023-11-02T08:48:48.067 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:48:48.067 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:48.067 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.068 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:48.068 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:7115", 2023-11-02T08:48:48.068 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 0 2023-11-02T08:48:48.068 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.068 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.069 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.069 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:7115/0", 2023-11-02T08:48:48.069 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:7115/0", 2023-11-02T08:48:48.069 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 0, 2023-11-02T08:48:48.069 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 0, 2023-11-02T08:48:48.070 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_location": "{}" 2023-11-02T08:48:48.070 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.071 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.071 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.071 INFO:tasks.workunit.client.0.smithi116.stdout: "quorum": [ 2023-11-02T08:48:48.071 INFO:tasks.workunit.client.0.smithi116.stdout: 0 2023-11-02T08:48:48.072 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:48.072 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap": { 2023-11-02T08:48:48.072 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 51, 2023-11-02T08:48:48.072 INFO:tasks.workunit.client.0.smithi116.stdout: "fsid": "0c9a372f-cec7-47d9-a309-8248453aabeb", 2023-11-02T08:48:48.072 INFO:tasks.workunit.client.0.smithi116.stdout: "created": "2023-11-02T08:44:34.942253+0000", 2023-11-02T08:48:48.073 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:48:12.999213+0000", 2023-11-02T08:48:48.073 INFO:tasks.workunit.client.0.smithi116.stdout: "last_up_change": "2023-11-02T08:48:12.612066+0000", 2023-11-02T08:48:48.074 INFO:tasks.workunit.client.0.smithi116.stdout: "last_in_change": "2023-11-02T08:47:24.817974+0000", 2023-11-02T08:48:48.074 INFO:tasks.workunit.client.0.smithi116.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-11-02T08:48:48.074 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_num": 5800384, 2023-11-02T08:48:48.075 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_set": [ 2023-11-02T08:48:48.075 INFO:tasks.workunit.client.0.smithi116.stdout: "nodown", 2023-11-02T08:48:48.075 INFO:tasks.workunit.client.0.smithi116.stdout: "noin", 2023-11-02T08:48:48.075 INFO:tasks.workunit.client.0.smithi116.stdout: "noout", 2023-11-02T08:48:48.076 INFO:tasks.workunit.client.0.smithi116.stdout: "pglog_hardlimit", 2023-11-02T08:48:48.076 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snapdirs", 2023-11-02T08:48:48.076 INFO:tasks.workunit.client.0.smithi116.stdout: "recovery_deletes", 2023-11-02T08:48:48.076 INFO:tasks.workunit.client.0.smithi116.stdout: "sortbitwise" 2023-11-02T08:48:48.076 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:48.078 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_version": 12, 2023-11-02T08:48:48.078 INFO:tasks.workunit.client.0.smithi116.stdout: "full_ratio": 0.99000000953674316, 2023-11-02T08:48:48.078 INFO:tasks.workunit.client.0.smithi116.stdout: "backfillfull_ratio": 0.99000000953674316, 2023-11-02T08:48:48.079 INFO:tasks.workunit.client.0.smithi116.stdout: "nearfull_ratio": 0.85000002384185791, 2023-11-02T08:48:48.079 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_snapshot": "", 2023-11-02T08:48:48.079 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_max": 1, 2023-11-02T08:48:48.079 INFO:tasks.workunit.client.0.smithi116.stdout: "max_osd": 3, 2023-11-02T08:48:48.079 INFO:tasks.workunit.client.0.smithi116.stdout: "require_min_compat_client": "luminous", 2023-11-02T08:48:48.080 INFO:tasks.workunit.client.0.smithi116.stdout: "min_compat_client": "jewel", 2023-11-02T08:48:48.080 INFO:tasks.workunit.client.0.smithi116.stdout: "require_osd_release": "reef", 2023-11-02T08:48:48.080 INFO:tasks.workunit.client.0.smithi116.stdout: "allow_crimson": false, 2023-11-02T08:48:48.080 INFO:tasks.workunit.client.0.smithi116.stdout: "pools": [ 2023-11-02T08:48:48.080 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.080 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": 1, 2023-11-02T08:48:48.081 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_name": "test", 2023-11-02T08:48:48.081 INFO:tasks.workunit.client.0.smithi116.stdout: "create_time": "2023-11-02T08:46:17.984400+0000", 2023-11-02T08:48:48.081 INFO:tasks.workunit.client.0.smithi116.stdout: "flags": 16389, 2023-11-02T08:48:48.082 INFO:tasks.workunit.client.0.smithi116.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2023-11-02T08:48:48.082 INFO:tasks.workunit.client.0.smithi116.stdout: "type": 3, 2023-11-02T08:48:48.082 INFO:tasks.workunit.client.0.smithi116.stdout: "size": 3, 2023-11-02T08:48:48.083 INFO:tasks.workunit.client.0.smithi116.stdout: "min_size": 2, 2023-11-02T08:48:48.083 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_rule": 1, 2023-11-02T08:48:48.083 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_count": 0, 2023-11-02T08:48:48.084 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_target": 0, 2023-11-02T08:48:48.084 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_barrier": 0, 2023-11-02T08:48:48.084 INFO:tasks.workunit.client.0.smithi116.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-11-02T08:48:48.084 INFO:tasks.workunit.client.0.smithi116.stdout: "object_hash": 2, 2023-11-02T08:48:48.085 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_autoscale_mode": "off", 2023-11-02T08:48:48.085 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num": 1, 2023-11-02T08:48:48.085 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_placement_num": 1, 2023-11-02T08:48:48.085 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_placement_num_target": 1, 2023-11-02T08:48:48.085 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num_target": 1, 2023-11-02T08:48:48.086 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_num_pending": 1, 2023-11-02T08:48:48.086 INFO:tasks.workunit.client.0.smithi116.stdout: "last_pg_merge_meta": { 2023-11-02T08:48:48.086 INFO:tasks.workunit.client.0.smithi116.stdout: "source_pgid": "0.0", 2023-11-02T08:48:48.086 INFO:tasks.workunit.client.0.smithi116.stdout: "ready_epoch": 0, 2023-11-02T08:48:48.086 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_started": 0, 2023-11-02T08:48:48.086 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_clean": 0, 2023-11-02T08:48:48.087 INFO:tasks.workunit.client.0.smithi116.stdout: "source_version": "0'0", 2023-11-02T08:48:48.087 INFO:tasks.workunit.client.0.smithi116.stdout: "target_version": "0'0" 2023-11-02T08:48:48.087 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.087 INFO:tasks.workunit.client.0.smithi116.stdout: "last_change": "30", 2023-11-02T08:48:48.087 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend": "0", 2023-11-02T08:48:48.088 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend_prenautilus": "0", 2023-11-02T08:48:48.088 INFO:tasks.workunit.client.0.smithi116.stdout: "last_force_op_resend_preluminous": "0", 2023-11-02T08:48:48.088 INFO:tasks.workunit.client.0.smithi116.stdout: "auid": 0, 2023-11-02T08:48:48.088 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_mode": "pool", 2023-11-02T08:48:48.088 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_seq": 1, 2023-11-02T08:48:48.089 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_epoch": 30, 2023-11-02T08:48:48.089 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_snaps": [ 2023-11-02T08:48:48.089 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.089 INFO:tasks.workunit.client.0.smithi116.stdout: "snapid": 1, 2023-11-02T08:48:48.089 INFO:tasks.workunit.client.0.smithi116.stdout: "stamp": "2023-11-02T08:47:01.329439+0000", 2023-11-02T08:48:48.090 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "snap1" 2023-11-02T08:48:48.090 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.090 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:48.090 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_snaps": "[]", 2023-11-02T08:48:48.090 INFO:tasks.workunit.client.0.smithi116.stdout: "quota_max_bytes": 0, 2023-11-02T08:48:48.091 INFO:tasks.workunit.client.0.smithi116.stdout: "quota_max_objects": 0, 2023-11-02T08:48:48.091 INFO:tasks.workunit.client.0.smithi116.stdout: "tiers": [], 2023-11-02T08:48:48.091 INFO:tasks.workunit.client.0.smithi116.stdout: "tier_of": -1, 2023-11-02T08:48:48.091 INFO:tasks.workunit.client.0.smithi116.stdout: "read_tier": -1, 2023-11-02T08:48:48.091 INFO:tasks.workunit.client.0.smithi116.stdout: "write_tier": -1, 2023-11-02T08:48:48.092 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_mode": "none", 2023-11-02T08:48:48.092 INFO:tasks.workunit.client.0.smithi116.stdout: "target_max_bytes": 0, 2023-11-02T08:48:48.092 INFO:tasks.workunit.client.0.smithi116.stdout: "target_max_objects": 0, 2023-11-02T08:48:48.092 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-11-02T08:48:48.092 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-11-02T08:48:48.092 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_target_full_ratio_micro": 800000, 2023-11-02T08:48:48.093 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_min_flush_age": 0, 2023-11-02T08:48:48.093 INFO:tasks.workunit.client.0.smithi116.stdout: "cache_min_evict_age": 0, 2023-11-02T08:48:48.093 INFO:tasks.workunit.client.0.smithi116.stdout: "erasure_code_profile": "myprofile", 2023-11-02T08:48:48.094 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_params": { 2023-11-02T08:48:48.094 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "none" 2023-11-02T08:48:48.094 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.094 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_period": 0, 2023-11-02T08:48:48.094 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_count": 0, 2023-11-02T08:48:48.095 INFO:tasks.workunit.client.0.smithi116.stdout: "use_gmt_hitset": true, 2023-11-02T08:48:48.095 INFO:tasks.workunit.client.0.smithi116.stdout: "min_read_recency_for_promote": 0, 2023-11-02T08:48:48.095 INFO:tasks.workunit.client.0.smithi116.stdout: "min_write_recency_for_promote": 0, 2023-11-02T08:48:48.095 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_grade_decay_rate": 0, 2023-11-02T08:48:48.095 INFO:tasks.workunit.client.0.smithi116.stdout: "hit_set_search_last_n": 0, 2023-11-02T08:48:48.096 INFO:tasks.workunit.client.0.smithi116.stdout: "grade_table": [], 2023-11-02T08:48:48.096 INFO:tasks.workunit.client.0.smithi116.stdout: "stripe_width": 8192, 2023-11-02T08:48:48.096 INFO:tasks.workunit.client.0.smithi116.stdout: "expected_num_objects": 0, 2023-11-02T08:48:48.096 INFO:tasks.workunit.client.0.smithi116.stdout: "fast_read": false, 2023-11-02T08:48:48.096 INFO:tasks.workunit.client.0.smithi116.stdout: "options": {}, 2023-11-02T08:48:48.097 INFO:tasks.workunit.client.0.smithi116.stdout: "application_metadata": {} 2023-11-02T08:48:48.097 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.097 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:48.097 INFO:tasks.workunit.client.0.smithi116.stdout: "osds": [ 2023-11-02T08:48:48.097 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.098 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:48:48.098 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "5f38bac9-3e89-4df4-9a41-799f865c97c8", 2023-11-02T08:48:48.098 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 1, 2023-11-02T08:48:48.098 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:48:48.099 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:48:48.099 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:48:48.099 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 37, 2023-11-02T08:48:48.099 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 43, 2023-11-02T08:48:48.099 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 46, 2023-11-02T08:48:48.100 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 50, 2023-11-02T08:48:48.100 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 45, 2023-11-02T08:48:48.100 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:48:48.100 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:48:48.100 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:48.100 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.101 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:48.101 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6802", 2023-11-02T08:48:48.101 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 137550 2023-11-02T08:48:48.101 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.101 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.102 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:48.102 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6803", 2023-11-02T08:48:48.102 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 137550 2023-11-02T08:48:48.102 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.102 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.103 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.103 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:48:48.103 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:48.103 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.103 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:48.103 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6804", 2023-11-02T08:48:48.104 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 137550 2023-11-02T08:48:48.104 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.104 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.104 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:48.104 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6805", 2023-11-02T08:48:48.105 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 137550 2023-11-02T08:48:48.105 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.105 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.105 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.105 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:48:48.106 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:48.106 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.106 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:48.106 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6808", 2023-11-02T08:48:48.106 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 137550 2023-11-02T08:48:48.107 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.107 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.107 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:48.107 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6809", 2023-11-02T08:48:48.107 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 137550 2023-11-02T08:48:48.108 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.108 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.108 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.108 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:48:48.108 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:48.108 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.109 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:48.109 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6806", 2023-11-02T08:48:48.109 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 137550 2023-11-02T08:48:48.109 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.109 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.110 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:48.110 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6807", 2023-11-02T08:48:48.110 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 137550 2023-11-02T08:48:48.110 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.110 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.111 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.111 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6803/137550", 2023-11-02T08:48:48.111 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6805/137550", 2023-11-02T08:48:48.111 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6809/137550", 2023-11-02T08:48:48.111 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6807/137550", 2023-11-02T08:48:48.112 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:48:48.112 INFO:tasks.workunit.client.0.smithi116.stdout: "exists", 2023-11-02T08:48:48.112 INFO:tasks.workunit.client.0.smithi116.stdout: "up" 2023-11-02T08:48:48.112 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.112 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.113 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.113 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:48:48.113 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "00de077e-ecc3-49b3-b8fc-83cfdd948349", 2023-11-02T08:48:48.113 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 0, 2023-11-02T08:48:48.113 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:48:48.113 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:48:48.114 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:48:48.114 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 0, 2023-11-02T08:48:48.114 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 0, 2023-11-02T08:48:48.114 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 10, 2023-11-02T08:48:48.114 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 25, 2023-11-02T08:48:48.115 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 32, 2023-11-02T08:48:48.115 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:48:48.115 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:48:48.116 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:48.116 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.116 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:48.116 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6810", 2023-11-02T08:48:48.116 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 128752 2023-11-02T08:48:48.116 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.117 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.117 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:48.117 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6811", 2023-11-02T08:48:48.117 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 128752 2023-11-02T08:48:48.118 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.118 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.118 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.118 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:48:48.118 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:48.119 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.119 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:48.119 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6812", 2023-11-02T08:48:48.119 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 128752 2023-11-02T08:48:48.119 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.119 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.120 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:48.120 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6813", 2023-11-02T08:48:48.120 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 128752 2023-11-02T08:48:48.120 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.120 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.121 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.121 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:48:48.121 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:48.121 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.121 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:48.121 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6816", 2023-11-02T08:48:48.122 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 128752 2023-11-02T08:48:48.122 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.122 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.122 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:48.122 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6817", 2023-11-02T08:48:48.123 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 128752 2023-11-02T08:48:48.123 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.123 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.123 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.123 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:48:48.123 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:48.124 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.124 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:48.124 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6814", 2023-11-02T08:48:48.124 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 128752 2023-11-02T08:48:48.125 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.125 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.125 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:48.125 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6815", 2023-11-02T08:48:48.125 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 128752 2023-11-02T08:48:48.126 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.126 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.126 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.126 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6811/128752", 2023-11-02T08:48:48.126 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6813/128752", 2023-11-02T08:48:48.127 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6817/128752", 2023-11-02T08:48:48.127 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6815/128752", 2023-11-02T08:48:48.127 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:48:48.127 INFO:tasks.workunit.client.0.smithi116.stdout: "exists" 2023-11-02T08:48:48.127 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.127 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.128 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.128 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:48:48.128 INFO:tasks.workunit.client.0.smithi116.stdout: "uuid": "c3024fa8-7c9d-4674-b3d4-e8b9122765a2", 2023-11-02T08:48:48.128 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 1, 2023-11-02T08:48:48.128 INFO:tasks.workunit.client.0.smithi116.stdout: "in": 1, 2023-11-02T08:48:48.129 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 1, 2023-11-02T08:48:48.129 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_affinity": 1, 2023-11-02T08:48:48.129 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_begin": 39, 2023-11-02T08:48:48.129 INFO:tasks.workunit.client.0.smithi116.stdout: "last_clean_end": 43, 2023-11-02T08:48:48.129 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 50, 2023-11-02T08:48:48.129 INFO:tasks.workunit.client.0.smithi116.stdout: "up_thru": 0, 2023-11-02T08:48:48.130 INFO:tasks.workunit.client.0.smithi116.stdout: "down_at": 49, 2023-11-02T08:48:48.130 INFO:tasks.workunit.client.0.smithi116.stdout: "lost_at": 0, 2023-11-02T08:48:48.130 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addrs": { 2023-11-02T08:48:48.130 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:48.130 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.131 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:48.131 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6810", 2023-11-02T08:48:48.131 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 138221 2023-11-02T08:48:48.131 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.131 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.131 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:48.132 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6811", 2023-11-02T08:48:48.132 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 138221 2023-11-02T08:48:48.132 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.132 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.133 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.133 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addrs": { 2023-11-02T08:48:48.133 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:48.133 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.133 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:48.134 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6812", 2023-11-02T08:48:48.134 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 138221 2023-11-02T08:48:48.134 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.134 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.134 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:48.135 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6813", 2023-11-02T08:48:48.135 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 138221 2023-11-02T08:48:48.135 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.135 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.135 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.136 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addrs": { 2023-11-02T08:48:48.136 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:48.136 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.136 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:48.137 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6816", 2023-11-02T08:48:48.137 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 138221 2023-11-02T08:48:48.137 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.137 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.138 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:48.138 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6817", 2023-11-02T08:48:48.138 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 138221 2023-11-02T08:48:48.138 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.138 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.139 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.139 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addrs": { 2023-11-02T08:48:48.139 INFO:tasks.workunit.client.0.smithi116.stdout: "addrvec": [ 2023-11-02T08:48:48.140 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.140 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v2", 2023-11-02T08:48:48.140 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6814", 2023-11-02T08:48:48.141 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 138221 2023-11-02T08:48:48.141 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.141 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.141 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "v1", 2023-11-02T08:48:48.141 INFO:tasks.workunit.client.0.smithi116.stdout: "addr": "127.0.0.1:6815", 2023-11-02T08:48:48.142 INFO:tasks.workunit.client.0.smithi116.stdout: "nonce": 138221 2023-11-02T08:48:48.142 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.142 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.142 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.142 INFO:tasks.workunit.client.0.smithi116.stdout: "public_addr": "127.0.0.1:6811/138221", 2023-11-02T08:48:48.142 INFO:tasks.workunit.client.0.smithi116.stdout: "cluster_addr": "127.0.0.1:6813/138221", 2023-11-02T08:48:48.143 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_back_addr": "127.0.0.1:6817/138221", 2023-11-02T08:48:48.143 INFO:tasks.workunit.client.0.smithi116.stdout: "heartbeat_front_addr": "127.0.0.1:6815/138221", 2023-11-02T08:48:48.143 INFO:tasks.workunit.client.0.smithi116.stdout: "state": [ 2023-11-02T08:48:48.143 INFO:tasks.workunit.client.0.smithi116.stdout: "exists", 2023-11-02T08:48:48.143 INFO:tasks.workunit.client.0.smithi116.stdout: "up" 2023-11-02T08:48:48.144 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.144 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.144 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:48.144 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_xinfo": [ 2023-11-02T08:48:48.144 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.144 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:48:48.145 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:48:10.344341+0000", 2023-11-02T08:48:48.145 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:48:48.145 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:48:48.145 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:48:48.145 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:48:48.146 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:45:06.084398+0000", 2023-11-02T08:48:48.146 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:48:48.146 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.146 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.146 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:48:48.147 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:47:12.926534+0000", 2023-11-02T08:48:48.147 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:48:48.147 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:48:48.147 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:48:48.147 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:48:48.148 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:45:39.287740+0000", 2023-11-02T08:48:48.148 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:48:48.148 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.148 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.148 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:48:48.148 INFO:tasks.workunit.client.0.smithi116.stdout: "down_stamp": "2023-11-02T08:48:12.463414+0000", 2023-11-02T08:48:48.149 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_probability": 0, 2023-11-02T08:48:48.149 INFO:tasks.workunit.client.0.smithi116.stdout: "laggy_interval": 0, 2023-11-02T08:48:48.149 INFO:tasks.workunit.client.0.smithi116.stdout: "features": 4540138322906710015, 2023-11-02T08:48:48.150 INFO:tasks.workunit.client.0.smithi116.stdout: "old_weight": 0, 2023-11-02T08:48:48.150 INFO:tasks.workunit.client.0.smithi116.stdout: "last_purged_snaps_scrub": "2023-11-02T08:46:13.065366+0000", 2023-11-02T08:48:48.150 INFO:tasks.workunit.client.0.smithi116.stdout: "dead_epoch": 0 2023-11-02T08:48:48.150 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.150 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:48.151 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap": [], 2023-11-02T08:48:48.151 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap_items": [], 2023-11-02T08:48:48.151 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_upmap_primaries": [], 2023-11-02T08:48:48.151 INFO:tasks.workunit.client.0.smithi116.stdout: "pg_temp": [], 2023-11-02T08:48:48.151 INFO:tasks.workunit.client.0.smithi116.stdout: "primary_temp": [], 2023-11-02T08:48:48.151 INFO:tasks.workunit.client.0.smithi116.stdout: "blocklist": {}, 2023-11-02T08:48:48.152 INFO:tasks.workunit.client.0.smithi116.stdout: "range_blocklist": {}, 2023-11-02T08:48:48.152 INFO:tasks.workunit.client.0.smithi116.stdout: "erasure_code_profiles": { 2023-11-02T08:48:48.152 INFO:tasks.workunit.client.0.smithi116.stdout: "default": { 2023-11-02T08:48:48.152 INFO:tasks.workunit.client.0.smithi116.stdout: "k": "2", 2023-11-02T08:48:48.152 INFO:tasks.workunit.client.0.smithi116.stdout: "m": "2", 2023-11-02T08:48:48.153 INFO:tasks.workunit.client.0.smithi116.stdout: "plugin": "jerasure", 2023-11-02T08:48:48.153 INFO:tasks.workunit.client.0.smithi116.stdout: "technique": "reed_sol_van" 2023-11-02T08:48:48.153 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.153 INFO:tasks.workunit.client.0.smithi116.stdout: "myprofile": { 2023-11-02T08:48:48.153 INFO:tasks.workunit.client.0.smithi116.stdout: "crush-device-class": "", 2023-11-02T08:48:48.154 INFO:tasks.workunit.client.0.smithi116.stdout: "crush-failure-domain": "osd", 2023-11-02T08:48:48.154 INFO:tasks.workunit.client.0.smithi116.stdout: "crush-root": "default", 2023-11-02T08:48:48.154 INFO:tasks.workunit.client.0.smithi116.stdout: "jerasure-per-chunk-alignment": "false", 2023-11-02T08:48:48.154 INFO:tasks.workunit.client.0.smithi116.stdout: "k": "2", 2023-11-02T08:48:48.154 INFO:tasks.workunit.client.0.smithi116.stdout: "m": "1", 2023-11-02T08:48:48.155 INFO:tasks.workunit.client.0.smithi116.stdout: "plugin": "jerasure", 2023-11-02T08:48:48.155 INFO:tasks.workunit.client.0.smithi116.stdout: "technique": "reed_sol_van", 2023-11-02T08:48:48.155 INFO:tasks.workunit.client.0.smithi116.stdout: "w": "8" 2023-11-02T08:48:48.155 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.155 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.155 INFO:tasks.workunit.client.0.smithi116.stdout: "removed_snaps_queue": [], 2023-11-02T08:48:48.156 INFO:tasks.workunit.client.0.smithi116.stdout: "new_removed_snaps": [], 2023-11-02T08:48:48.156 INFO:tasks.workunit.client.0.smithi116.stdout: "new_purged_snaps": [], 2023-11-02T08:48:48.156 INFO:tasks.workunit.client.0.smithi116.stdout: "crush_node_flags": {}, 2023-11-02T08:48:48.156 INFO:tasks.workunit.client.0.smithi116.stdout: "device_class_flags": {}, 2023-11-02T08:48:48.156 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode": { 2023-11-02T08:48:48.157 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode_enabled": false, 2023-11-02T08:48:48.157 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_bucket_count": 0, 2023-11-02T08:48:48.157 INFO:tasks.workunit.client.0.smithi116.stdout: "degraded_stretch_mode": 0, 2023-11-02T08:48:48.157 INFO:tasks.workunit.client.0.smithi116.stdout: "recovering_stretch_mode": 0, 2023-11-02T08:48:48.157 INFO:tasks.workunit.client.0.smithi116.stdout: "stretch_mode_bucket": 0 2023-11-02T08:48:48.158 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.158 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.158 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_metadata": [ 2023-11-02T08:48:48.158 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.158 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:48:48.159 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:48:48.159 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6804/137550,v1:127.0.0.1:6805/137550]", 2023-11-02T08:48:48.159 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:48:48.159 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:48:48.159 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:48:48.160 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:48:48.160 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:48:48.160 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:48:48.160 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:48:48.160 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:48:48.161 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:48:48.161 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:48:48.161 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2023-11-02T08:48:48.161 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:48:48.161 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:48:48.162 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:48:48.162 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:48:48.162 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:48:48.162 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:48:48.162 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:48:48.163 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:48:48.163 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:48:48.163 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:48:48.163 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:44:40.798381Z", 2023-11-02T08:48:48.163 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:48:48.164 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:48:48.164 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:48:48.164 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:48:48.164 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:48:48.164 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:48:48.165 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:48:48.165 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6802/137550,v1:127.0.0.1:6803/137550]", 2023-11-02T08:48:48.165 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:48:48.165 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/137550,v1:127.0.0.1:6809/137550]", 2023-11-02T08:48:48.165 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/137550,v1:127.0.0.1:6807/137550]", 2023-11-02T08:48:48.166 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:48:48.166 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:48:48.166 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:48:48.166 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:48:48.166 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:48:48.167 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:48:48.167 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:48:48.167 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:48:48.167 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:48:48.167 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/0", 2023-11-02T08:48:48.167 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:48:48.168 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:48:48.168 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:48:48.168 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.168 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.168 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:48:48.169 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:48:48.169 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6812/128752,v1:127.0.0.1:6813/128752]", 2023-11-02T08:48:48.169 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:48:48.169 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:48:48.170 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:48:48.170 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:48:48.170 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:48:48.170 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:48:48.170 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:48:48.171 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:48:48.171 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:48:48.171 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:48:48.171 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2023-11-02T08:48:48.171 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:48:48.171 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:48:48.172 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:48:48.172 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:48:48.172 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:48:48.172 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:48:48.172 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:48:48.173 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:48:48.173 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:48:48.173 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:48:48.173 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:45:13.085521Z", 2023-11-02T08:48:48.173 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:48:48.174 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:48:48.174 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:48:48.174 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:48:48.174 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:48:48.174 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:48:48.175 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:48:48.175 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6810/128752,v1:127.0.0.1:6811/128752]", 2023-11-02T08:48:48.175 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:48:48.175 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/128752,v1:127.0.0.1:6817/128752]", 2023-11-02T08:48:48.175 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/128752,v1:127.0.0.1:6815/128752]", 2023-11-02T08:48:48.176 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:48:48.176 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:48:48.176 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:48:48.176 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:48:48.176 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:48:48.177 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:48:48.177 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:48:48.177 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:48:48.177 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:48:48.177 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/1", 2023-11-02T08:48:48.177 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:48:48.178 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:48:48.178 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:48:48.178 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.178 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.178 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:48:48.179 INFO:tasks.workunit.client.0.smithi116.stdout: "arch": "x86_64", 2023-11-02T08:48:48.179 INFO:tasks.workunit.client.0.smithi116.stdout: "back_addr": "[v2:127.0.0.1:6812/138221,v1:127.0.0.1:6813/138221]", 2023-11-02T08:48:48.179 INFO:tasks.workunit.client.0.smithi116.stdout: "back_iface": "", 2023-11-02T08:48:48.179 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs": "1", 2023-11-02T08:48:48.180 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_db": "0", 2023-11-02T08:48:48.180 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_dedicated_wal": "0", 2023-11-02T08:48:48.180 INFO:tasks.workunit.client.0.smithi116.stdout: "bluefs_single_shared_device": "1", 2023-11-02T08:48:48.180 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_access_mode": "file", 2023-11-02T08:48:48.180 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_block_size": "4096", 2023-11-02T08:48:48.181 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_devices": "sda", 2023-11-02T08:48:48.181 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-11-02T08:48:48.181 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-11-02T08:48:48.181 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2023-11-02T08:48:48.181 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_rotational": "1", 2023-11-02T08:48:48.182 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_size": "107374182400", 2023-11-02T08:48:48.182 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_support_discard": "0", 2023-11-02T08:48:48.182 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_bdev_type": "hdd", 2023-11-02T08:48:48.182 INFO:tasks.workunit.client.0.smithi116.stdout: "bluestore_min_alloc_size": "4096", 2023-11-02T08:48:48.182 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_release": "reef", 2023-11-02T08:48:48.183 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:48:48.183 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_short": "18.2.0-1159-gb5a9486b", 2023-11-02T08:48:48.183 INFO:tasks.workunit.client.0.smithi116.stdout: "ceph_version_when_created": "ceph version 18.2.0-1159-gb5a9486b (b5a9486b4a6f144f83c51b7ec33111a2c86dc081) reef (stable)", 2023-11-02T08:48:48.184 INFO:tasks.workunit.client.0.smithi116.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-11-02T08:48:48.184 INFO:tasks.workunit.client.0.smithi116.stdout: "created_at": "2023-11-02T08:45:46.537324Z", 2023-11-02T08:48:48.184 INFO:tasks.workunit.client.0.smithi116.stdout: "default_device_class": "hdd", 2023-11-02T08:48:48.184 INFO:tasks.workunit.client.0.smithi116.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QJB5", 2023-11-02T08:48:48.184 INFO:tasks.workunit.client.0.smithi116.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-11-02T08:48:48.184 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": "sda", 2023-11-02T08:48:48.185 INFO:tasks.workunit.client.0.smithi116.stdout: "distro": "centos", 2023-11-02T08:48:48.185 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_description": "CentOS Stream 9", 2023-11-02T08:48:48.185 INFO:tasks.workunit.client.0.smithi116.stdout: "distro_version": "9", 2023-11-02T08:48:48.185 INFO:tasks.workunit.client.0.smithi116.stdout: "front_addr": "[v2:127.0.0.1:6810/138221,v1:127.0.0.1:6811/138221]", 2023-11-02T08:48:48.185 INFO:tasks.workunit.client.0.smithi116.stdout: "front_iface": "", 2023-11-02T08:48:48.186 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/138221,v1:127.0.0.1:6817/138221]", 2023-11-02T08:48:48.186 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/138221,v1:127.0.0.1:6815/138221]", 2023-11-02T08:48:48.186 INFO:tasks.workunit.client.0.smithi116.stdout: "hostname": "smithi116", 2023-11-02T08:48:48.186 INFO:tasks.workunit.client.0.smithi116.stdout: "journal_rotational": "1", 2023-11-02T08:48:48.186 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Oct 19 18:39:36 UTC 2023", 2023-11-02T08:48:48.187 INFO:tasks.workunit.client.0.smithi116.stdout: "kernel_version": "5.14.0-378.el9.x86_64", 2023-11-02T08:48:48.187 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_swap_kb": "0", 2023-11-02T08:48:48.187 INFO:tasks.workunit.client.0.smithi116.stdout: "mem_total_kb": "32488704", 2023-11-02T08:48:48.187 INFO:tasks.workunit.client.0.smithi116.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-11-02T08:48:48.187 INFO:tasks.workunit.client.0.smithi116.stdout: "objectstore_numa_unknown_devices": "sda", 2023-11-02T08:48:48.188 INFO:tasks.workunit.client.0.smithi116.stdout: "os": "Linux", 2023-11-02T08:48:48.188 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_data": "td/divergent-priors/2", 2023-11-02T08:48:48.188 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_objectstore": "bluestore", 2023-11-02T08:48:48.188 INFO:tasks.workunit.client.0.smithi116.stdout: "osdspec_affinity": "", 2023-11-02T08:48:48.188 INFO:tasks.workunit.client.0.smithi116.stdout: "rotational": "1" 2023-11-02T08:48:48.189 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.189 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:48.189 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_clean_epochs": { 2023-11-02T08:48:48.189 INFO:tasks.workunit.client.0.smithi116.stdout: "min_last_epoch_clean": 10, 2023-11-02T08:48:48.189 INFO:tasks.workunit.client.0.smithi116.stdout: "last_epoch_clean": { 2023-11-02T08:48:48.190 INFO:tasks.workunit.client.0.smithi116.stdout: "per_pool": [ 2023-11-02T08:48:48.190 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.190 INFO:tasks.workunit.client.0.smithi116.stdout: "poolid": 1, 2023-11-02T08:48:48.190 INFO:tasks.workunit.client.0.smithi116.stdout: "floor": 26 2023-11-02T08:48:48.190 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.191 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.191 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.191 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_epochs": [ 2023-11-02T08:48:48.191 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.191 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:48:48.192 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 47 2023-11-02T08:48:48.192 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.192 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.192 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:48:48.192 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 10 2023-11-02T08:48:48.192 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.193 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.193 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:48:48.193 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 50 2023-11-02T08:48:48.193 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.193 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.194 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.194 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_first_committed": 1, 2023-11-02T08:48:48.194 INFO:tasks.workunit.client.0.smithi116.stdout: "osdmap_last_committed": 51, 2023-11-02T08:48:48.194 INFO:tasks.workunit.client.0.smithi116.stdout: "crushmap": { 2023-11-02T08:48:48.195 INFO:tasks.workunit.client.0.smithi116.stdout: "devices": [ 2023-11-02T08:48:48.195 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.195 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:48:48.195 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.0", 2023-11-02T08:48:48.195 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:48:48.196 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.196 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.196 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:48:48.196 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.1", 2023-11-02T08:48:48.196 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:48:48.197 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.197 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.197 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:48:48.197 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd.2", 2023-11-02T08:48:48.197 INFO:tasks.workunit.client.0.smithi116.stdout: "class": "hdd" 2023-11-02T08:48:48.198 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.198 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:48.198 INFO:tasks.workunit.client.0.smithi116.stdout: "types": [ 2023-11-02T08:48:48.198 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.198 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 0, 2023-11-02T08:48:48.199 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "osd" 2023-11-02T08:48:48.199 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.199 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.199 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:48:48.199 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "host" 2023-11-02T08:48:48.200 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.200 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.200 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 2, 2023-11-02T08:48:48.200 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "chassis" 2023-11-02T08:48:48.200 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.200 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.201 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 3, 2023-11-02T08:48:48.201 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "rack" 2023-11-02T08:48:48.201 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.201 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.201 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 4, 2023-11-02T08:48:48.202 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "row" 2023-11-02T08:48:48.202 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.202 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.202 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 5, 2023-11-02T08:48:48.202 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "pdu" 2023-11-02T08:48:48.203 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.203 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.203 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 6, 2023-11-02T08:48:48.203 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "pod" 2023-11-02T08:48:48.203 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.203 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.204 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 7, 2023-11-02T08:48:48.204 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "room" 2023-11-02T08:48:48.204 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.204 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.204 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 8, 2023-11-02T08:48:48.205 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "datacenter" 2023-11-02T08:48:48.205 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.205 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.205 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 9, 2023-11-02T08:48:48.205 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "zone" 2023-11-02T08:48:48.205 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.206 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.206 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 10, 2023-11-02T08:48:48.206 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "region" 2023-11-02T08:48:48.206 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.207 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.207 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:48:48.207 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "root" 2023-11-02T08:48:48.207 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.207 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:48.208 INFO:tasks.workunit.client.0.smithi116.stdout: "buckets": [ 2023-11-02T08:48:48.208 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.208 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -1, 2023-11-02T08:48:48.208 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "default", 2023-11-02T08:48:48.208 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:48:48.208 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "root", 2023-11-02T08:48:48.209 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:48:48.209 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:48:48.209 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:48:48.209 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:48:48.209 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.209 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -3, 2023-11-02T08:48:48.210 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:48:48.210 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:48:48.210 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.210 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.210 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.211 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.211 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -2, 2023-11-02T08:48:48.211 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "default~hdd", 2023-11-02T08:48:48.211 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 11, 2023-11-02T08:48:48.212 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "root", 2023-11-02T08:48:48.212 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:48:48.212 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:48:48.212 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:48:48.212 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:48:48.213 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.213 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -4, 2023-11-02T08:48:48.213 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:48:48.213 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:48:48.214 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.214 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.214 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.214 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.214 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -3, 2023-11-02T08:48:48.215 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "smithi116", 2023-11-02T08:48:48.215 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:48:48.215 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "host", 2023-11-02T08:48:48.215 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:48:48.215 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:48:48.215 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:48:48.216 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:48:48.216 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.216 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:48:48.216 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:48:48.216 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:48:48.217 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.217 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.217 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:48:48.217 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:48:48.217 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 1 2023-11-02T08:48:48.218 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.218 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.218 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:48:48.218 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:48:48.218 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 2 2023-11-02T08:48:48.219 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.219 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.219 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.219 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.219 INFO:tasks.workunit.client.0.smithi116.stdout: "id": -4, 2023-11-02T08:48:48.220 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "smithi116~hdd", 2023-11-02T08:48:48.220 INFO:tasks.workunit.client.0.smithi116.stdout: "type_id": 1, 2023-11-02T08:48:48.220 INFO:tasks.workunit.client.0.smithi116.stdout: "type_name": "host", 2023-11-02T08:48:48.220 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 19206, 2023-11-02T08:48:48.220 INFO:tasks.workunit.client.0.smithi116.stdout: "alg": "straw2", 2023-11-02T08:48:48.221 INFO:tasks.workunit.client.0.smithi116.stdout: "hash": "rjenkins1", 2023-11-02T08:48:48.221 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T08:48:48.221 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.221 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 0, 2023-11-02T08:48:48.221 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:48:48.222 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 0 2023-11-02T08:48:48.222 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.222 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.222 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2023-11-02T08:48:48.222 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:48:48.223 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 1 2023-11-02T08:48:48.223 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.223 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.223 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2023-11-02T08:48:48.223 INFO:tasks.workunit.client.0.smithi116.stdout: "weight": 6402, 2023-11-02T08:48:48.224 INFO:tasks.workunit.client.0.smithi116.stdout: "pos": 2 2023-11-02T08:48:48.224 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.224 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.224 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.224 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:48.225 INFO:tasks.workunit.client.0.smithi116.stdout: "rules": [ 2023-11-02T08:48:48.225 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.225 INFO:tasks.workunit.client.0.smithi116.stdout: "rule_id": 0, 2023-11-02T08:48:48.225 INFO:tasks.workunit.client.0.smithi116.stdout: "rule_name": "replicated_rule", 2023-11-02T08:48:48.225 INFO:tasks.workunit.client.0.smithi116.stdout: "type": 1, 2023-11-02T08:48:48.226 INFO:tasks.workunit.client.0.smithi116.stdout: "steps": [ 2023-11-02T08:48:48.226 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.226 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "take", 2023-11-02T08:48:48.226 INFO:tasks.workunit.client.0.smithi116.stdout: "item": -1, 2023-11-02T08:48:48.226 INFO:tasks.workunit.client.0.smithi116.stdout: "item_name": "default" 2023-11-02T08:48:48.227 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.227 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.227 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "choose_firstn", 2023-11-02T08:48:48.227 INFO:tasks.workunit.client.0.smithi116.stdout: "num": 0, 2023-11-02T08:48:48.227 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "osd" 2023-11-02T08:48:48.227 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.228 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.228 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "emit" 2023-11-02T08:48:48.228 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.228 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.228 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.229 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.229 INFO:tasks.workunit.client.0.smithi116.stdout: "rule_id": 1, 2023-11-02T08:48:48.229 INFO:tasks.workunit.client.0.smithi116.stdout: "rule_name": "test", 2023-11-02T08:48:48.229 INFO:tasks.workunit.client.0.smithi116.stdout: "type": 3, 2023-11-02T08:48:48.229 INFO:tasks.workunit.client.0.smithi116.stdout: "steps": [ 2023-11-02T08:48:48.230 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.230 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "set_chooseleaf_tries", 2023-11-02T08:48:48.230 INFO:tasks.workunit.client.0.smithi116.stdout: "num": 5 2023-11-02T08:48:48.230 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.230 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.230 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "set_choose_tries", 2023-11-02T08:48:48.231 INFO:tasks.workunit.client.0.smithi116.stdout: "num": 100 2023-11-02T08:48:48.231 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.231 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.231 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "take", 2023-11-02T08:48:48.231 INFO:tasks.workunit.client.0.smithi116.stdout: "item": -1, 2023-11-02T08:48:48.232 INFO:tasks.workunit.client.0.smithi116.stdout: "item_name": "default" 2023-11-02T08:48:48.232 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.232 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.232 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "choose_indep", 2023-11-02T08:48:48.232 INFO:tasks.workunit.client.0.smithi116.stdout: "num": 0, 2023-11-02T08:48:48.232 INFO:tasks.workunit.client.0.smithi116.stdout: "type": "osd" 2023-11-02T08:48:48.233 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.233 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.233 INFO:tasks.workunit.client.0.smithi116.stdout: "op": "emit" 2023-11-02T08:48:48.233 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.233 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T08:48:48.233 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.234 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:48.234 INFO:tasks.workunit.client.0.smithi116.stdout: "tunables": { 2023-11-02T08:48:48.234 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_local_tries": 0, 2023-11-02T08:48:48.234 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_local_fallback_tries": 0, 2023-11-02T08:48:48.234 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_total_tries": 50, 2023-11-02T08:48:48.235 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_descend_once": 1, 2023-11-02T08:48:48.235 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_vary_r": 1, 2023-11-02T08:48:48.235 INFO:tasks.workunit.client.0.smithi116.stdout: "chooseleaf_stable": 1, 2023-11-02T08:48:48.235 INFO:tasks.workunit.client.0.smithi116.stdout: "straw_calc_version": 1, 2023-11-02T08:48:48.236 INFO:tasks.workunit.client.0.smithi116.stdout: "allowed_bucket_algs": 54, 2023-11-02T08:48:48.236 INFO:tasks.workunit.client.0.smithi116.stderr:report 335540463 2023-11-02T08:48:48.237 INFO:tasks.workunit.client.0.smithi116.stdout: "profile": "jewel", 2023-11-02T08:48:48.237 INFO:tasks.workunit.client.0.smithi116.stdout: "optimal_tunables": 1, 2023-11-02T08:48:48.237 INFO:tasks.workunit.client.0.smithi116.stdout: "legacy_tunables": 0, 2023-11-02T08:48:48.237 INFO:tasks.workunit.client.0.smithi116.stdout: "minimum_required_version": "jewel", 2023-11-02T08:48:48.237 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables": 1, 2023-11-02T08:48:48.238 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables2": 1, 2023-11-02T08:48:48.238 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v2_rules": 1, 2023-11-02T08:48:48.238 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables3": 1, 2023-11-02T08:48:48.238 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v3_rules": 0, 2023-11-02T08:48:48.238 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v4_buckets": 1, 2023-11-02T08:48:48.239 INFO:tasks.workunit.client.0.smithi116.stdout: "require_feature_tunables5": 1, 2023-11-02T08:48:48.239 INFO:tasks.workunit.client.0.smithi116.stdout: "has_v5_rules": 0 2023-11-02T08:48:48.239 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.239 INFO:tasks.workunit.client.0.smithi116.stdout: "choose_args": {} 2023-11-02T08:48:48.239 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.240 INFO:tasks.workunit.client.0.smithi116.stdout: "fsmap": { 2023-11-02T08:48:48.240 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 1, 2023-11-02T08:48:48.240 INFO:tasks.workunit.client.0.smithi116.stdout: "default_fscid": -1, 2023-11-02T08:48:48.240 INFO:tasks.workunit.client.0.smithi116.stdout: "compat": { 2023-11-02T08:48:48.240 INFO:tasks.workunit.client.0.smithi116.stdout: "compat": {}, 2023-11-02T08:48:48.241 INFO:tasks.workunit.client.0.smithi116.stdout: "ro_compat": {}, 2023-11-02T08:48:48.241 INFO:tasks.workunit.client.0.smithi116.stdout: "incompat": { 2023-11-02T08:48:48.241 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_1": "base v0.20", 2023-11-02T08:48:48.241 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_2": "client writeable ranges", 2023-11-02T08:48:48.241 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_3": "default file layouts on dirs", 2023-11-02T08:48:48.242 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_4": "dir inode in separate object", 2023-11-02T08:48:48.242 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_5": "mds uses versioned encoding", 2023-11-02T08:48:48.242 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_6": "dirfrag is stored in omap", 2023-11-02T08:48:48.242 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_8": "no anchor table", 2023-11-02T08:48:48.242 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_9": "file layout v2", 2023-11-02T08:48:48.243 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_10": "snaprealm v2" 2023-11-02T08:48:48.243 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.243 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.243 INFO:tasks.workunit.client.0.smithi116.stdout: "feature_flags": { 2023-11-02T08:48:48.243 INFO:tasks.workunit.client.0.smithi116.stdout: "enable_multiple": true, 2023-11-02T08:48:48.244 INFO:tasks.workunit.client.0.smithi116.stdout: "ever_enabled_multiple": true 2023-11-02T08:48:48.244 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.244 INFO:tasks.workunit.client.0.smithi116.stdout: "standbys": [], 2023-11-02T08:48:48.244 INFO:tasks.workunit.client.0.smithi116.stdout: "filesystems": [] 2023-11-02T08:48:48.244 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.245 INFO:tasks.workunit.client.0.smithi116.stdout: "mdsmap_first_committed": 1, 2023-11-02T08:48:48.245 INFO:tasks.workunit.client.0.smithi116.stdout: "mdsmap_last_committed": 1, 2023-11-02T08:48:48.245 INFO:tasks.workunit.client.0.smithi116.stdout: "auth": { 2023-11-02T08:48:48.245 INFO:tasks.workunit.client.0.smithi116.stdout: "first_committed": 1, 2023-11-02T08:48:48.245 INFO:tasks.workunit.client.0.smithi116.stdout: "last_committed": 5, 2023-11-02T08:48:48.246 INFO:tasks.workunit.client.0.smithi116.stdout: "num_secrets": 3 2023-11-02T08:48:48.246 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.246 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg": 1, 2023-11-02T08:48:48.246 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_active": 1, 2023-11-02T08:48:48.246 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_unknown": 0, 2023-11-02T08:48:48.247 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osd": 3, 2023-11-02T08:48:48.247 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_sum": { 2023-11-02T08:48:48.247 INFO:tasks.workunit.client.0.smithi116.stdout: "stat_sum": { 2023-11-02T08:48:48.247 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes": 48430, 2023-11-02T08:48:48.247 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects": 101, 2023-11-02T08:48:48.248 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_clones": 1, 2023-11-02T08:48:48.248 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_copies": 303, 2023-11-02T08:48:48.248 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing_on_primary": 0, 2023-11-02T08:48:48.248 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing": 0, 2023-11-02T08:48:48.248 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_degraded": 101, 2023-11-02T08:48:48.249 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_misplaced": 0, 2023-11-02T08:48:48.249 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_unfound": 0, 2023-11-02T08:48:48.249 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_dirty": 101, 2023-11-02T08:48:48.249 INFO:tasks.workunit.client.0.smithi116.stdout: "num_whiteouts": 0, 2023-11-02T08:48:48.249 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read": 0, 2023-11-02T08:48:48.250 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read_kb": 0, 2023-11-02T08:48:48.250 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write": 101, 2023-11-02T08:48:48.250 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write_kb": 101, 2023-11-02T08:48:48.253 INFO:tasks.workunit.client.0.smithi116.stdout: "num_scrub_errors": 0, 2023-11-02T08:48:48.253 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shallow_scrub_errors": 0, 2023-11-02T08:48:48.254 INFO:tasks.workunit.client.0.smithi116.stdout: "num_deep_scrub_errors": 0, 2023-11-02T08:48:48.254 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_recovered": 0, 2023-11-02T08:48:48.254 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_recovered": 0, 2023-11-02T08:48:48.254 INFO:tasks.workunit.client.0.smithi116.stdout: "num_keys_recovered": 0, 2023-11-02T08:48:48.255 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_omap": 0, 2023-11-02T08:48:48.255 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_hit_set_archive": 0, 2023-11-02T08:48:48.255 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_hit_set_archive": 0, 2023-11-02T08:48:48.255 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush": 0, 2023-11-02T08:48:48.255 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_kb": 0, 2023-11-02T08:48:48.255 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict": 0, 2023-11-02T08:48:48.256 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_kb": 0, 2023-11-02T08:48:48.256 INFO:tasks.workunit.client.0.smithi116.stdout: "num_promote": 0, 2023-11-02T08:48:48.256 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_high": 0, 2023-11-02T08:48:48.256 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_low": 0, 2023-11-02T08:48:48.256 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_some": 0, 2023-11-02T08:48:48.257 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_full": 0, 2023-11-02T08:48:48.257 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_pinned": 0, 2023-11-02T08:48:48.257 INFO:tasks.workunit.client.0.smithi116.stdout: "num_legacy_snapsets": 0, 2023-11-02T08:48:48.257 INFO:tasks.workunit.client.0.smithi116.stdout: "num_large_omap_objects": 0, 2023-11-02T08:48:48.257 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_manifest": 0, 2023-11-02T08:48:48.258 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_bytes": 0, 2023-11-02T08:48:48.258 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_keys": 0, 2023-11-02T08:48:48.258 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_repaired": 0 2023-11-02T08:48:48.258 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.258 INFO:tasks.workunit.client.0.smithi116.stdout: "store_stats": { 2023-11-02T08:48:48.258 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 0, 2023-11-02T08:48:48.259 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 0, 2023-11-02T08:48:48.259 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:48:48.259 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 0, 2023-11-02T08:48:48.259 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 0, 2023-11-02T08:48:48.259 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:48:48.260 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:48:48.260 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:48:48.260 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 0, 2023-11-02T08:48:48.260 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 0 2023-11-02T08:48:48.260 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.261 INFO:tasks.workunit.client.0.smithi116.stdout: "log_size": 18, 2023-11-02T08:48:48.261 INFO:tasks.workunit.client.0.smithi116.stdout: "ondisk_log_size": 18, 2023-11-02T08:48:48.261 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 3, 2023-11-02T08:48:48.261 INFO:tasks.workunit.client.0.smithi116.stdout: "acting": 3, 2023-11-02T08:48:48.261 INFO:tasks.workunit.client.0.smithi116.stdout: "num_store_stats": 0 2023-11-02T08:48:48.262 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.262 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_sum": { 2023-11-02T08:48:48.262 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 0, 2023-11-02T08:48:48.262 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 0, 2023-11-02T08:48:48.262 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pgs": 3, 2023-11-02T08:48:48.263 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osds": 3, 2023-11-02T08:48:48.263 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_osds": 3, 2023-11-02T08:48:48.263 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_omap_osds": 3, 2023-11-02T08:48:48.263 INFO:tasks.workunit.client.0.smithi116.stdout: "kb": 314572800, 2023-11-02T08:48:48.263 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used": 84624, 2023-11-02T08:48:48.263 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_data": 2288, 2023-11-02T08:48:48.264 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_omap": 5, 2023-11-02T08:48:48.264 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_meta": 82298, 2023-11-02T08:48:48.264 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_avail": 314488176, 2023-11-02T08:48:48.264 INFO:tasks.workunit.client.0.smithi116.stdout: "statfs": { 2023-11-02T08:48:48.264 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 322122547200, 2023-11-02T08:48:48.265 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 322035892224, 2023-11-02T08:48:48.265 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:48:48.265 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 2342912, 2023-11-02T08:48:48.265 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 1600121, 2023-11-02T08:48:48.266 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:48:48.266 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:48:48.266 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:48:48.266 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 5278, 2023-11-02T08:48:48.266 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 84274018 2023-11-02T08:48:48.267 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.267 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_peers": [], 2023-11-02T08:48:48.267 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_trim_queue_len": 0, 2023-11-02T08:48:48.267 INFO:tasks.workunit.client.0.smithi116.stdout: "num_snap_trimming": 0, 2023-11-02T08:48:48.267 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shards_repaired": 0, 2023-11-02T08:48:48.267 INFO:tasks.workunit.client.0.smithi116.stdout: "op_queue_age_hist": { 2023-11-02T08:48:48.268 INFO:tasks.workunit.client.0.smithi116.stdout: "histogram": [], 2023-11-02T08:48:48.268 INFO:tasks.workunit.client.0.smithi116.stdout: "upper_bound": 1 2023-11-02T08:48:48.268 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.268 INFO:tasks.workunit.client.0.smithi116.stdout: "perf_stat": { 2023-11-02T08:48:48.268 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ms": 0, 2023-11-02T08:48:48.269 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ms": 0, 2023-11-02T08:48:48.269 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ns": 0, 2023-11-02T08:48:48.269 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ns": 0 2023-11-02T08:48:48.269 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.269 INFO:tasks.workunit.client.0.smithi116.stdout: "alerts": [], 2023-11-02T08:48:48.270 INFO:tasks.workunit.client.0.smithi116.stdout: "network_ping_times": [] 2023-11-02T08:48:48.270 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.270 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_sum_by_class": { 2023-11-02T08:48:48.270 INFO:tasks.workunit.client.0.smithi116.stdout: "hdd": { 2023-11-02T08:48:48.270 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 0, 2023-11-02T08:48:48.270 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 0, 2023-11-02T08:48:48.271 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pgs": 3, 2023-11-02T08:48:48.271 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osds": 3, 2023-11-02T08:48:48.271 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_osds": 3, 2023-11-02T08:48:48.271 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_omap_osds": 3, 2023-11-02T08:48:48.271 INFO:tasks.workunit.client.0.smithi116.stdout: "kb": 314572800, 2023-11-02T08:48:48.272 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used": 84624, 2023-11-02T08:48:48.272 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_data": 2288, 2023-11-02T08:48:48.272 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_omap": 5, 2023-11-02T08:48:48.272 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_meta": 82298, 2023-11-02T08:48:48.272 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_avail": 314488176, 2023-11-02T08:48:48.273 INFO:tasks.workunit.client.0.smithi116.stdout: "statfs": { 2023-11-02T08:48:48.273 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 322122547200, 2023-11-02T08:48:48.273 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 322035892224, 2023-11-02T08:48:48.273 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:48:48.273 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 2342912, 2023-11-02T08:48:48.274 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 1600121, 2023-11-02T08:48:48.274 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:48:48.274 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:48:48.274 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:48:48.274 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 5278, 2023-11-02T08:48:48.275 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 84274018 2023-11-02T08:48:48.275 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.275 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_peers": [], 2023-11-02T08:48:48.275 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_trim_queue_len": 0, 2023-11-02T08:48:48.275 INFO:tasks.workunit.client.0.smithi116.stdout: "num_snap_trimming": 0, 2023-11-02T08:48:48.276 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shards_repaired": 0, 2023-11-02T08:48:48.276 INFO:tasks.workunit.client.0.smithi116.stdout: "op_queue_age_hist": { 2023-11-02T08:48:48.276 INFO:tasks.workunit.client.0.smithi116.stdout: "histogram": [], 2023-11-02T08:48:48.276 INFO:tasks.workunit.client.0.smithi116.stdout: "upper_bound": 1 2023-11-02T08:48:48.276 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.277 INFO:tasks.workunit.client.0.smithi116.stdout: "perf_stat": { 2023-11-02T08:48:48.277 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ms": 0, 2023-11-02T08:48:48.277 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ms": 0, 2023-11-02T08:48:48.277 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ns": 0, 2023-11-02T08:48:48.277 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ns": 0 2023-11-02T08:48:48.278 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.278 INFO:tasks.workunit.client.0.smithi116.stdout: "alerts": [], 2023-11-02T08:48:48.278 INFO:tasks.workunit.client.0.smithi116.stdout: "network_ping_times": [] 2023-11-02T08:48:48.278 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.278 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.279 INFO:tasks.workunit.client.0.smithi116.stdout: "pool_stats": [ 2023-11-02T08:48:48.279 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.279 INFO:tasks.workunit.client.0.smithi116.stdout: "poolid": 1, 2023-11-02T08:48:48.279 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg": 1, 2023-11-02T08:48:48.279 INFO:tasks.workunit.client.0.smithi116.stdout: "stat_sum": { 2023-11-02T08:48:48.280 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes": 48430, 2023-11-02T08:48:48.280 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects": 101, 2023-11-02T08:48:48.280 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_clones": 1, 2023-11-02T08:48:48.280 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_copies": 303, 2023-11-02T08:48:48.280 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing_on_primary": 0, 2023-11-02T08:48:48.281 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing": 0, 2023-11-02T08:48:48.281 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_degraded": 101, 2023-11-02T08:48:48.281 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_misplaced": 0, 2023-11-02T08:48:48.281 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_unfound": 0, 2023-11-02T08:48:48.281 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_dirty": 101, 2023-11-02T08:48:48.281 INFO:tasks.workunit.client.0.smithi116.stdout: "num_whiteouts": 0, 2023-11-02T08:48:48.282 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read": 0, 2023-11-02T08:48:48.282 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read_kb": 0, 2023-11-02T08:48:48.282 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write": 101, 2023-11-02T08:48:48.282 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write_kb": 101, 2023-11-02T08:48:48.283 INFO:tasks.workunit.client.0.smithi116.stdout: "num_scrub_errors": 0, 2023-11-02T08:48:48.283 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shallow_scrub_errors": 0, 2023-11-02T08:48:48.283 INFO:tasks.workunit.client.0.smithi116.stdout: "num_deep_scrub_errors": 0, 2023-11-02T08:48:48.283 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_recovered": 0, 2023-11-02T08:48:48.284 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_recovered": 0, 2023-11-02T08:48:48.284 INFO:tasks.workunit.client.0.smithi116.stdout: "num_keys_recovered": 0, 2023-11-02T08:48:48.284 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_omap": 0, 2023-11-02T08:48:48.284 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_hit_set_archive": 0, 2023-11-02T08:48:48.284 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_hit_set_archive": 0, 2023-11-02T08:48:48.285 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush": 0, 2023-11-02T08:48:48.285 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_kb": 0, 2023-11-02T08:48:48.285 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict": 0, 2023-11-02T08:48:48.285 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_kb": 0, 2023-11-02T08:48:48.285 INFO:tasks.workunit.client.0.smithi116.stdout: "num_promote": 0, 2023-11-02T08:48:48.285 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_high": 0, 2023-11-02T08:48:48.286 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_low": 0, 2023-11-02T08:48:48.286 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_some": 0, 2023-11-02T08:48:48.286 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_full": 0, 2023-11-02T08:48:48.286 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_pinned": 0, 2023-11-02T08:48:48.286 INFO:tasks.workunit.client.0.smithi116.stdout: "num_legacy_snapsets": 0, 2023-11-02T08:48:48.287 INFO:tasks.workunit.client.0.smithi116.stdout: "num_large_omap_objects": 0, 2023-11-02T08:48:48.287 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_manifest": 0, 2023-11-02T08:48:48.287 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_bytes": 0, 2023-11-02T08:48:48.287 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_keys": 0, 2023-11-02T08:48:48.287 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_repaired": 0 2023-11-02T08:48:48.288 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.288 INFO:tasks.workunit.client.0.smithi116.stdout: "store_stats": { 2023-11-02T08:48:48.288 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 0, 2023-11-02T08:48:48.288 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 0, 2023-11-02T08:48:48.288 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2023-11-02T08:48:48.289 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 1241088, 2023-11-02T08:48:48.289 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 1241088, 2023-11-02T08:48:48.289 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2023-11-02T08:48:48.289 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2023-11-02T08:48:48.289 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2023-11-02T08:48:48.289 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 0, 2023-11-02T08:48:48.290 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 0 2023-11-02T08:48:48.290 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.290 INFO:tasks.workunit.client.0.smithi116.stdout: "log_size": 18, 2023-11-02T08:48:48.290 INFO:tasks.workunit.client.0.smithi116.stdout: "ondisk_log_size": 18, 2023-11-02T08:48:48.290 INFO:tasks.workunit.client.0.smithi116.stdout: "up": 3, 2023-11-02T08:48:48.291 INFO:tasks.workunit.client.0.smithi116.stdout: "acting": 3, 2023-11-02T08:48:48.291 INFO:tasks.workunit.client.0.smithi116.stdout: "num_store_stats": 3 2023-11-02T08:48:48.291 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.291 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:48.292 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_stats": [ 2023-11-02T08:48:48.292 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.292 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:48:48.292 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 197568495626 2023-11-02T08:48:48.292 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.292 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.293 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:48:48.293 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 42949672988 2023-11-02T08:48:48.293 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.293 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.293 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:48:48.294 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 214748364808 2023-11-02T08:48:48.294 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.294 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:48.294 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_by_state": [ 2023-11-02T08:48:48.294 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.294 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "active+undersized+degraded", 2023-11-02T08:48:48.295 INFO:tasks.workunit.client.0.smithi116.stdout: "num": 1 2023-11-02T08:48:48.295 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.295 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:48.295 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pg_by_osd": [ 2023-11-02T08:48:48.295 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.296 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2147483647, 2023-11-02T08:48:48.296 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 0, 2023-11-02T08:48:48.296 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 1, 2023-11-02T08:48:48.296 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:48:48.296 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.297 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.297 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 2, 2023-11-02T08:48:48.297 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 0, 2023-11-02T08:48:48.297 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 1, 2023-11-02T08:48:48.297 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:48:48.298 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.298 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.298 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2023-11-02T08:48:48.298 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 1, 2023-11-02T08:48:48.298 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 1, 2023-11-02T08:48:48.299 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:48:48.299 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.299 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.299 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2023-11-02T08:48:48.299 INFO:tasks.workunit.client.0.smithi116.stdout: "num_primary_pg": 0, 2023-11-02T08:48:48.299 INFO:tasks.workunit.client.0.smithi116.stdout: "num_acting_pg": 0, 2023-11-02T08:48:48.300 INFO:tasks.workunit.client.0.smithi116.stdout: "num_up_not_acting_pg": 0 2023-11-02T08:48:48.300 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.300 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:48.300 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snaps": [ 2023-11-02T08:48:48.300 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T08:48:48.301 INFO:tasks.workunit.client.0.smithi116.stdout: "pool": 1, 2023-11-02T08:48:48.301 INFO:tasks.workunit.client.0.smithi116.stdout: "purged_snaps": {} 2023-11-02T08:48:48.301 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.301 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T08:48:48.301 INFO:tasks.workunit.client.0.smithi116.stdout: "servicemap": { 2023-11-02T08:48:48.301 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": 5, 2023-11-02T08:48:48.302 INFO:tasks.workunit.client.0.smithi116.stdout: "modified": "2023-11-02T08:47:15.584380+0000", 2023-11-02T08:48:48.302 INFO:tasks.workunit.client.0.smithi116.stdout: "services": {} 2023-11-02T08:48:48.302 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T08:48:48.302 INFO:tasks.workunit.client.0.smithi116.stdout: "mgrstat_first_committed": 1, 2023-11-02T08:48:48.302 INFO:tasks.workunit.client.0.smithi116.stdout: "mgrstat_last_committed": 116, 2023-11-02T08:48:48.303 INFO:tasks.workunit.client.0.smithi116.stdout: "logm_first_committed": 1, 2023-11-02T08:48:48.303 INFO:tasks.workunit.client.0.smithi116.stdout: "logm_last_committed": 471, 2023-11-02T08:48:48.303 INFO:tasks.workunit.client.0.smithi116.stdout: "paxos": { 2023-11-02T08:48:48.303 INFO:tasks.workunit.client.0.smithi116.stdout: "first_committed": 1, 2023-11-02T08:48:48.303 INFO:tasks.workunit.client.0.smithi116.stdout: "last_committed": 688, 2023-11-02T08:48:48.304 INFO:tasks.workunit.client.0.smithi116.stdout: "last_pn": 0, 2023-11-02T08:48:48.304 INFO:tasks.workunit.client.0.smithi116.stdout: "accepted_pn": 0 2023-11-02T08:48:48.304 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T08:48:48.304 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T08:48:48.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2023-11-02T08:48:48.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2023-11-02T08:48:48.305 INFO:tasks.workunit.client.0.smithi116.stdout:delay recovery 2023-11-02T08:48:48.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2023-11-02T08:48:48.407 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:48:48.407 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 101 0 101 0 0 48430 0 0 18 84 18 active+undersized+degraded 2023-11-02T08:48:13.308491+0000 43'102 51:139 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2023-11-02T08:46:18.062675+0000 0'0 2023-11-02T08:46:18.062675+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:17:03.240704+0000 0 0 2023-11-02T08:48:48.407 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:48:48.408 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:48:48.408 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:48:48.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2023-11-02T08:48:48.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2023-11-02T08:48:48.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-11-02T08:48:48.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-11-02T08:48:48.420 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:48:48.420 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:48:48.420 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:48:48.420 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.0.asok 2023-11-02T08:48:48.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2023-11-02T08:48:48.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.0.asok set_recovery_delay 100000 2023-11-02T08:48:48.500 INFO:tasks.workunit.client.0.smithi116.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2023-11-02T08:48:48.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2023-11-02T08:48:48.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2023-11-02T08:48:48.509 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-11-02T08:48:48.509 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-11-02T08:48:48.509 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:48:48.509 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:48:48.509 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:48:48.510 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.2.asok 2023-11-02T08:48:48.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2023-11-02T08:48:48.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.2.asok set_recovery_delay 100000 2023-11-02T08:48:48.588 INFO:tasks.workunit.client.0.smithi116.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2023-11-02T08:48:48.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2023-11-02T08:48:48.596 INFO:tasks.workunit.client.0.smithi116.stdout:revive divergent 1 2023-11-02T08:48:48.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2023-11-02T08:48:48.953 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:48:48.953 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 101 0 101 0 0 48430 0 0 18 84 18 active+undersized+degraded 2023-11-02T08:48:13.308491+0000 43'102 51:139 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2023-11-02T08:46:18.062675+0000 0'0 2023-11-02T08:46:18.062675+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:17:03.240704+0000 0 0 2023-11-02T08:48:48.953 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:48:48.954 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:48:48.954 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:48:48.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2023-11-02T08:48:49.583 INFO:tasks.workunit.client.0.smithi116.stderr:noup is set 2023-11-02T08:48:49.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2023-11-02T08:48:49.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2023-11-02T08:48:49.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:48:49.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2023-11-02T08:48:49.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:48:49.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2023-11-02T08:48:49.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:48:49.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:48:49.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:48:49.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:48:49.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2023-11-02T08:48:49.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2023-11-02T08:48:49.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:48:49.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:48:49.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:48:49.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:48:49.602 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:48:49.602 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:48:49.603 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:48:49.603 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:48:49.603 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:48:49.604 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:48:49.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:48:49.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:48:49.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:48:49.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:48:49.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:48:49.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:48:49.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:48:49.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:48:49.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:48:49.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:48:49.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2023-11-02T08:48:49.606 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T08:48:49.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2023-11-02T08:48:49.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:48:49.609 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2023-11-02T08:48:49.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2023-11-02T08:48:49.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:48:49.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:48:49.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:48:49.630 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:48:49.627+0000 7f8a2308a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:48:49.631 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:48:49.629+0000 7f8a2308a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:48:49.632 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:48:49.630+0000 7f8a2308a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:48:50.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2023-11-02T08:48:50.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2023-11-02T08:48:50.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:48:50.144+0000 7f8a2308a500 -1 Falling back to public interface 2023-11-02T08:48:52.477 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:48:52.474+0000 7f8a2308a500 -1 osd.1 30 log_to_monitors true 2023-11-02T08:48:55.057 INFO:tasks.workunit.client.0.smithi116.stdout:delay recovery divergent 2023-11-02T08:48:55.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2023-11-02T08:48:55.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2023-11-02T08:48:55.423 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:48:55.423 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 101 0 101 0 0 48430 0 0 18 84 18 active+undersized+degraded 2023-11-02T08:48:13.308491+0000 43'102 52:141 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2023-11-02T08:46:18.062675+0000 0'0 2023-11-02T08:46:18.062675+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:17:03.240704+0000 0 0 2023-11-02T08:48:55.423 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:48:55.424 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:48:55.424 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:48:55.435 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2023-11-02T08:48:55.435 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-11-02T08:48:55.435 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-11-02T08:48:55.435 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:48:55.436 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:48:55.436 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:48:55.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85082/ceph-osd.1.asok 2023-11-02T08:48:55.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2023-11-02T08:48:55.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.85082/ceph-osd.1.asok set_recovery_delay 100000 2023-11-02T08:48:55.515 INFO:tasks.workunit.client.0.smithi116.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2023-11-02T08:48:55.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2023-11-02T08:48:56.086 INFO:tasks.workunit.client.0.smithi116.stderr:noup is unset 2023-11-02T08:48:56.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2023-11-02T08:48:56.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:48:56.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:48:56.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:48:56.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:48:56.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:48:56.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:48:56.099 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:48:56.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:48:56.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:48:56.546 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 46 up_thru 50 down_at 45 last_clean_interval [37,43) [v2:127.0.0.1:6802/137550,v1:127.0.0.1:6803/137550] [v2:127.0.0.1:6804/137550,v1:127.0.0.1:6805/137550] exists,up 5f38bac9-3e89-4df4-9a41-799f865c97c8 2023-11-02T08:48:56.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:48:56.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:48:56.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:48:56.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2023-11-02T08:48:56.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:48:56.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:48:56.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:48:56.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:48:56.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:48:56.549 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:48:56.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:48:56.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:48:56.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:48:57.035 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 56 up_thru 56 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/140264,v1:127.0.0.1:6819/140264] [v2:127.0.0.1:6820/140264,v1:127.0.0.1:6821/140264] exists,up 00de077e-ecc3-49b3-b8fc-83cfdd948349 2023-11-02T08:48:57.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:48:57.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:48:57.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:48:57.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2023-11-02T08:48:57.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:48:57.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T08:48:57.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:48:57.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:48:57.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:48:57.039 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:48:57.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:48:57.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:48:57.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:48:57.444 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 49 last_clean_interval [39,43) [v2:127.0.0.1:6810/138221,v1:127.0.0.1:6811/138221] [v2:127.0.0.1:6812/138221,v1:127.0.0.1:6813/138221] exists,up c3024fa8-7c9d-4674-b3d4-e8b9122765a2 2023-11-02T08:48:57.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:48:57.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:48:57.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:48:57.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2023-11-02T08:48:57.802 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:48:57.803 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 101 0 101 0 0 48430 0 0 18 84 18 active+undersized+degraded 2023-11-02T08:48:13.308491+0000 43'102 53:142 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2023-11-02T08:46:18.062675+0000 0'0 2023-11-02T08:46:18.062675+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:17:03.240704+0000 0 0 2023-11-02T08:48:57.803 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:48:57.803 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:48:57.803 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:48:57.813 INFO:tasks.workunit.client.0.smithi116.stdout:wait for peering 2023-11-02T08:48:57.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2023-11-02T08:48:57.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2023-11-02T08:48:58.174 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:48:58.174 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 101 0 101 0 0 48430 0 0 18 84 18 active+undersized+degraded 2023-11-02T08:48:13.308491+0000 43'102 53:142 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2023-11-02T08:46:18.062675+0000 0'0 2023-11-02T08:46:18.062675+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:17:03.240704+0000 0 0 2023-11-02T08:48:58.175 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:48:58.175 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:48:58.175 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:48:58.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2023-11-02T08:48:58.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2023-11-02T08:48:58.286 INFO:tasks.workunit.client.0.smithi116.stdout:killing divergent 1 2023-11-02T08:48:58.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2023-11-02T08:48:58.643 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:48:58.644 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 101 0 101 0 0 48430 0 0 18 84 18 active+undersized+degraded 2023-11-02T08:48:13.308491+0000 43'102 53:142 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2023-11-02T08:46:18.062675+0000 0'0 2023-11-02T08:46:18.062675+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:17:03.240704+0000 0 0 2023-11-02T08:48:58.644 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:48:58.644 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:48:58.644 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:48:58.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2023-11-02T08:48:58.654 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:48:58.655 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:48:58.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:48:58.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:48:58.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:48:58.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:48:58.760 INFO:tasks.workunit.client.0.smithi116.stdout:reviving divergent 1 2023-11-02T08:48:58.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2023-11-02T08:48:58.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2023-11-02T08:48:59.118 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:48:59.119 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 101 0 101 0 0 48430 0 0 18 84 18 active+undersized+degraded 2023-11-02T08:48:13.308491+0000 43'102 53:142 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2023-11-02T08:46:18.062675+0000 0'0 2023-11-02T08:46:18.062675+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:17:03.240704+0000 0 0 2023-11-02T08:48:59.119 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:48:59.119 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:48:59.119 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:48:59.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2023-11-02T08:48:59.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2023-11-02T08:48:59.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:48:59.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2023-11-02T08:48:59.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:48:59.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2023-11-02T08:48:59.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2023-11-02T08:48:59.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:48:59.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:48:59.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:48:59.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2023-11-02T08:48:59.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2023-11-02T08:48:59.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:48:59.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:48:59.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2023-11-02T08:48:59.135 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:48:59.135 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:48:59.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:48:59.137 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:48:59.137 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:48:59.137 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:48:59.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:48:59.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' 2023-11-02T08:48:59.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:48:59.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2023-11-02T08:48:59.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2023-11-02T08:48:59.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:48:59.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:48:59.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:48:59.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:48:59.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:48:59.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:48:59.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2023-11-02T08:48:59.140 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T08:48:59.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2023-11-02T08:48:59.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=0c9a372f-cec7-47d9-a309-8248453aabeb --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.85082/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:48:59.142 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2023-11-02T08:48:59.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2023-11-02T08:48:59.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:48:59.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:48:59.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:48:59.163 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:48:59.160+0000 7f68c0e2a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:48:59.165 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:48:59.161+0000 7f68c0e2a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:48:59.166 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:48:59.162+0000 7f68c0e2a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:48:59.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2023-11-02T08:48:59.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:48:59.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:48:59.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:48:59.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:48:59.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:48:59.511 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:48:59.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:48:59.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:48:59.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:48:59.681 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:48:59.677+0000 7f68c0e2a500 -1 Falling back to public interface 2023-11-02T08:48:59.895 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 56 up_thru 56 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/140264,v1:127.0.0.1:6819/140264] [v2:127.0.0.1:6820/140264,v1:127.0.0.1:6821/140264] exists,up 00de077e-ecc3-49b3-b8fc-83cfdd948349 2023-11-02T08:48:59.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:48:59.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:48:59.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:48:59.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2023-11-02T08:49:02.402 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:49:02.398+0000 7f68c0e2a500 -1 osd.1 57 log_to_monitors true 2023-11-02T08:49:19.897 INFO:tasks.workunit.client.0.smithi116.stdout:allowing recovery 2023-11-02T08:49:19.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2023-11-02T08:49:19.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2023-11-02T08:49:20.264 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:49:20.264 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 102 0 0 0 0 48913 0 0 13 90 13 active+clean 2023-11-02T08:49:04.185277+0000 57'103 62:153 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:46:18.062675+0000 0'0 2023-11-02T08:46:18.062675+0000 0 0 periodic scrub scheduled @ 2023-11-03T19:08:30.013169+0000 0 0 2023-11-02T08:49:20.264 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:49:20.265 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:49:20.265 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:49:20.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2023-11-02T08:49:20.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2023-11-02T08:49:20.385 INFO:tasks.workunit.client.0.smithi116.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2023-11-02T08:49:20.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2023-11-02T08:49:20.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2023-11-02T08:49:20.502 INFO:tasks.workunit.client.0.smithi116.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2023-11-02T08:49:20.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2023-11-02T08:49:20.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2023-11-02T08:49:20.622 INFO:tasks.workunit.client.0.smithi116.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2023-11-02T08:49:20.632 INFO:tasks.workunit.client.0.smithi116.stdout:reading divergent objects 2023-11-02T08:49:20.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2023-11-02T08:49:20.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2023-11-02T08:49:20.990 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T08:49:20.991 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 102 0 0 0 0 48913 0 0 13 90 13 active+clean 2023-11-02T08:49:04.185277+0000 57'103 62:153 [1,0,2] 1 [1,0,2] 1 0'0 2023-11-02T08:46:18.062675+0000 0'0 2023-11-02T08:46:18.062675+0000 0 0 periodic scrub scheduled @ 2023-11-03T19:08:30.013169+0000 0 0 2023-11-02T08:49:20.991 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T08:49:20.991 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T08:49:20.992 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T08:49:21.000 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2023-11-02T08:49:21.001 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2023-11-02T08:49:21.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2023-11-02T08:49:21.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2023-11-02T08:49:21.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2023-11-02T08:49:21.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2023-11-02T08:49:21.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2023-11-02T08:49:21.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2023-11-02T08:49:21.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2023-11-02T08:49:21.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2023-11-02T08:49:21.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2023-11-02T08:49:21.147 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2023-11-02T08:49:21.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log 2023-11-02T08:49:21.155 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2023-11-02T08:46:54.017218+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2023-11-02T08:49:21.156 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 20 _merge_object_divergent_entries: keeping 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2023-11-02T08:46:54.017218+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2023-11-02T08:49:21.156 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2023-11-02T08:49:21.156 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 29'99 first_divergent_update: 30'103 last_divergent_update: 30'103 2023-11-02T08:49:21.156 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2023-11-02T08:49:21.156 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2023-11-02T08:49:21.157 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4651.0:1 2023-11-02T08:47:01.518893+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:49:21.157 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 20 _merge_object_divergent_entries: keeping 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4651.0:1 2023-11-02T08:47:01.518893+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:49:21.157 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2023-11-02T08:49:21.157 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 29'99 first_divergent_update: 30'104 last_divergent_update: 30'104 2023-11-02T08:49:21.157 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2023-11-02T08:49:21.158 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2023-11-02T08:49:21.158 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 30'104 2023-11-02T08:49:21.158 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4651.0:1 2023-11-02T08:47:01.518893+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:49:21.158 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2023-11-02T08:49:21.158 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4642.0:1 2023-11-02T08:47:00.323337+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0,29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4648.0:1 2023-11-02T08:47:01.330058+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:49:21.159 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4642.0:1 2023-11-02T08:47:00.323337+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:49:21.159 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 20 _merge_object_divergent_entries: keeping 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4648.0:1 2023-11-02T08:47:01.330058+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:49:21.159 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2023-11-02T08:49:21.159 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'102 2023-11-02T08:49:21.159 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2023-11-02T08:49:21.159 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2023-11-02T08:49:21.160 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 30'104 2023-11-02T08:49:21.160 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4648.0:1 2023-11-02T08:47:01.330058+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2023-11-02T08:49:21.160 INFO:tasks.workunit.client.0.smithi116.stdout:td/divergent-priors/osd.1.log:2023-11-02T08:48:56.542+0000 7f8a06ffb640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2023-11-02T08:49:21.161 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2023-11-02T08:49:21.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log 2023-11-02T08:49:21.164 INFO:tasks.workunit.client.0.smithi116.stdout:success 2023-11-02T08:49:21.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2023-11-02T08:49:21.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2023-11-02T08:49:21.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2023-11-02T08:49:21.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2023-11-02T08:49:21.778 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test' does not exist 2023-11-02T08:49:21.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2023-11-02T08:49:21.790 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:49:21.790 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:49:21.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:49:21.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:49:21.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:49:22.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:49:22.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2023-11-02T08:49:22.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2023-11-02T08:49:22.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T08:49:22.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2023-11-02T08:49:22.099 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:49:22.099 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:49:22.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:49:22.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:49:22.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:49:22.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:49:22.104 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T08:49:22.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T08:49:22.105 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T08:49:22.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T08:49:22.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T08:49:22.106 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T08:49:22.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:49:22.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T08:49:22.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T08:49:22.107 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:49:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T08:49:22.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T08:49:22.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2023-11-02T08:49:22.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T08:49:22.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:49:22.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:49:22.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.85082 2023-11-02T08:49:22.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T08:49:22.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T08:49:22.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2023-11-02T08:49:22.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/divergent-priors 0 2023-11-02T08:49:22.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2023-11-02T08:49:22.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-11-02T08:49:22.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2023-11-02T08:49:22.133 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:49:22.133 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:49:22.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:49:22.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:49:22.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:49:22.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:49:22.135 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T08:49:22.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T08:49:22.135 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T08:49:22.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T08:49:22.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T08:49:22.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T08:49:22.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:49:22.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T08:49:22.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T08:49:22.139 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:49:22.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T08:49:22.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-11-02T08:49:22.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2023-11-02T08:49:22.142 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T08:49:22.143 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:49:22.143 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85082 2023-11-02T08:49:22.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.85082 2023-11-02T08:49:22.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T08:49:22.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T08:49:22.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2023-11-02T08:49:22.144 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-02T08:49:22.144 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-02T08:49:22.173 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2023-11-02T08:49:22.174 DEBUG:teuthology.orchestra.run.smithi116:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2023-11-02T08:49:22.233 INFO:tasks.workunit.client.0.smithi116.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-11-02T08:49:22.239 INFO:tasks.workunit.client.0.smithi116.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-11-02T08:49:22.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T08:49:22.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T08:49:22.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2023-11-02T08:49:22.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2023-11-02T08:49:22.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2023-11-02T08:49:22.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2023-11-02T08:49:22.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2023-11-02T08:49:22.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2023-11-02T08:49:22.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/ec-error-rollforward 2023-11-02T08:49:22.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2023-11-02T08:49:22.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2023-11-02T08:49:22.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2023-11-02T08:49:22.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2023-11-02T08:49:22.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2023-11-02T08:49:22.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2023-11-02T08:49:22.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none ' 2023-11-02T08:49:22.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2023-11-02T08:49:22.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2023-11-02T08:49:22.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2023-11-02T08:49:22.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2023-11-02T08:49:22.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2023-11-02T08:49:22.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2023-11-02T08:49:22.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2023-11-02T08:49:22.246 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2023-11-02T08:49:22.246 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-11-02T08:49:22.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2023-11-02T08:49:22.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2023-11-02T08:49:22.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2023-11-02T08:49:22.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2023-11-02T08:49:22.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2023-11-02T08:49:22.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2023-11-02T08:49:22.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T08:49:22.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2023-11-02T08:49:22.251 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:49:22.251 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:49:22.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:49:22.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:49:22.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:49:22.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:49:22.253 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T08:49:22.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T08:49:22.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T08:49:22.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T08:49:22.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T08:49:22.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T08:49:22.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:49:22.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T08:49:22.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T08:49:22.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:49:22.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T08:49:22.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T08:49:22.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2023-11-02T08:49:22.256 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T08:49:22.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:49:22.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141790 2023-11-02T08:49:22.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.141790 2023-11-02T08:49:22.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T08:49:22.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T08:49:22.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2023-11-02T08:49:22.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T08:49:22.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:49:22.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141790 2023-11-02T08:49:22.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.141790 2023-11-02T08:49:22.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T08:49:22.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-11-02T08:49:22.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-11-02T08:49:22.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T08:49:22.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2023-11-02T08:49:22.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2023-11-02T08:49:22.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2023-11-02T08:49:22.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2023-11-02T08:49:22.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2023-11-02T08:49:22.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T08:49:22.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T08:49:22.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T08:49:22.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2023-11-02T08:49:22.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2023-11-02T08:49:22.620 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T08:49:22.620 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:49:22.620 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:49:22.621 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:49:22.621 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:49:22.621 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141790 2023-11-02T08:49:22.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141790/$cluster-$name.asok' 2023-11-02T08:49:22.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T08:49:22.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T08:49:22.806 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T08:49:22.806 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T08:49:22.806 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T08:49:22.807 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T08:49:22.807 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T08:49:22.807 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T08:49:22.807 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T08:49:22.808 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T08:49:22.808 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:49:22.808 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:49:22.808 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141790 2023-11-02T08:49:22.808 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.141790/ceph-mon.a.asok 2023-11-02T08:49:22.809 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T08:49:22.809 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.141790/ceph-mon.a.asok config get fsid 2023-11-02T08:49:22.889 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T08:49:22.889 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T08:49:22.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T08:49:22.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T08:49:22.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T08:49:22.890 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T08:49:22.891 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T08:49:22.891 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T08:49:22.891 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:49:22.891 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:49:22.891 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141790 2023-11-02T08:49:22.891 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.141790/ceph-mon.a.asok 2023-11-02T08:49:22.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T08:49:22.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.141790/ceph-mon.a.asok config get mon_host 2023-11-02T08:49:22.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2023-11-02T08:49:22.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ec-error-rollforward 2023-11-02T08:49:22.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T08:49:22.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T08:49:22.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T08:49:22.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ec-error-rollforward/x 2023-11-02T08:49:22.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T08:49:23.828 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T08:49:23.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:49:23.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:49:23.829 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:49:23.829 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:49:23.829 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141790 2023-11-02T08:49:23.830 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141790/$cluster-$name.asok' 2023-11-02T08:49:23.830 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T08:49:23.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T08:49:23.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2023-11-02T08:49:23.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2023-11-02T08:49:23.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:49:23.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T08:49:23.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:49:23.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/0 2023-11-02T08:49:23.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2023-11-02T08:49:23.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:49:23.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:49:23.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:49:23.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2023-11-02T08:49:23.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2023-11-02T08:49:23.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:49:23.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:49:23.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2023-11-02T08:49:23.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:49:23.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:49:23.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:49:23.860 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:49:23.861 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:49:23.861 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141790 2023-11-02T08:49:23.861 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141790/$cluster-$name.asok' 2023-11-02T08:49:23.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' 2023-11-02T08:49:23.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:49:23.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:49:23.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:49:23.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2023-11-02T08:49:23.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2023-11-02T08:49:23.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:49:23.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:49:23.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:49:23.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:49:23.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:49:23.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:49:23.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/0 2023-11-02T08:49:23.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:49:23.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aaa308f4-98eb-42a0-9e17-452d97fb0f43 2023-11-02T08:49:23.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 aaa308f4-98eb-42a0-9e17-452d97fb0f43' 2023-11-02T08:49:23.866 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 aaa308f4-98eb-42a0-9e17-452d97fb0f43 2023-11-02T08:49:23.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:49:23.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTYkNls+7XMxAA0RaT6ZEo4P7bWyEhZ+qmiQ== 2023-11-02T08:49:23.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTYkNls+7XMxAA0RaT6ZEo4P7bWyEhZ+qmiQ=="}' 2023-11-02T08:49:23.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aaa308f4-98eb-42a0-9e17-452d97fb0f43 -i td/ec-error-rollforward/0/new.json 2023-11-02T08:49:24.230 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:49:24.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/0/new.json 2023-11-02T08:49:24.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTYkNls+7XMxAA0RaT6ZEo4P7bWyEhZ+qmiQ== --osd-uuid aaa308f4-98eb-42a0-9e17-452d97fb0f43 2023-11-02T08:49:24.267 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:49:24.264+0000 7f8390c10500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:49:24.268 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:49:24.266+0000 7f8390c10500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:49:24.269 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:49:24.267+0000 7f8390c10500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:49:24.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:49:24.267+0000 7f8390c10500 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2023-11-02T08:49:24.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:49:24.267+0000 7f8390c10500 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2023-11-02T08:49:24.270 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:49:24.267+0000 7f8390c10500 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2023-11-02T08:49:24.323 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:49:24.318+0000 7f8390c10500 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2023-11-02T08:49:29.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2023-11-02T08:49:29.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:49:29.654 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T08:49:29.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T08:49:29.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:49:30.173 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T08:49:30.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T08:49:30.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:49:30.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:49:30.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:49:30.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:49:30.197 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:49:30.194+0000 7f288fe16500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:49:30.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:49:30.200+0000 7f288fe16500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:49:30.203 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:49:30.201+0000 7f288fe16500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:49:30.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T08:49:30.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:49:30.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:49:30.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:49:30.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:49:30.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:49:30.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:49:30.609 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:49:30.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:49:30.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:49:30.734 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:49:30.731+0000 7f288fe16500 -1 Falling back to public interface 2023-11-02T08:49:30.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:49:31.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:49:31.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:49:31.975 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:49:31.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:49:31.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:49:31.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:49:32.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:49:33.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:49:33.043+0000 7f288fe16500 -1 osd.0 0 log_to_monitors true 2023-11-02T08:49:33.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:49:33.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:49:33.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:49:33.387 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:49:33.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:49:33.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:49:33.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:49:34.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:49:34.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:49:34.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:49:34.927 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:49:34.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:49:34.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:49:35.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:49:36.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:49:36.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:49:36.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:49:36.304 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:49:36.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:49:36.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:49:36.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:49:37.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:49:37.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:49:37.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:49:37.714 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:49:37.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:49:37.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:49:38.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:49:39.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:49:39.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:49:39.124 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:49:39.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:49:39.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:49:39.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:49:39.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:49:40.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:49:40.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:49:40.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:49:40.516 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:49:40.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:49:40.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:49:40.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:49:41.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:49:41.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:49:41.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:49:41.995 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:49:41.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:49:41.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:49:42.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:49:43.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:49:43.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:49:43.375 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:49:43.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:49:43.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:49:43.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:49:43.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:49:44.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:49:44.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:49:44.868 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:49:44.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:49:44.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:49:44.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:49:45.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:49:46.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:49:46.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:49:46.585 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:49:46.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:49:46.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:49:46.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:49:47.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:49:48.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:49:48.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:49:48.133 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:49:48.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:49:48.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:49:48.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:49:49.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:49:50.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:49:50.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:49:50.167 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:49:50.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:49:50.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:49:50.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:49:51.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:49:52.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:49:52.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:49:52.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:49:52.032 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:49:52.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:49:52.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:49:52.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:49:53.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:49:53.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:49:53.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:49:53.599 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:49:53.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:49:53.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:49:54.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:49:55.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:49:55.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:49:55.067 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:49:55.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:49:55.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:49:55.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:49:55.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:49:56.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:49:56.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:49:56.486 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T08:49:56.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T08:49:56.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:49:56.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:49:56.852 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/142874,v1:127.0.0.1:6803/142874] [v2:127.0.0.1:6804/142874,v1:127.0.0.1:6805/142874] exists,up aaa308f4-98eb-42a0-9e17-452d97fb0f43 2023-11-02T08:49:56.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:49:56.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:49:56.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:49:56.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2023-11-02T08:49:56.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2023-11-02T08:49:56.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:49:56.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-11-02T08:49:56.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:49:56.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/1 2023-11-02T08:49:56.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2023-11-02T08:49:56.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:49:56.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:49:56.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:49:56.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2023-11-02T08:49:56.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2023-11-02T08:49:56.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:49:56.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:49:56.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2023-11-02T08:49:56.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:49:56.867 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:49:56.867 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:49:56.867 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:49:56.868 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:49:56.868 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141790 2023-11-02T08:49:56.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141790/$cluster-$name.asok' 2023-11-02T08:49:56.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' 2023-11-02T08:49:56.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:49:56.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:49:56.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:49:56.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2023-11-02T08:49:56.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2023-11-02T08:49:56.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:49:56.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:49:56.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:49:56.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:49:56.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:49:56.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:49:56.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/1 2023-11-02T08:49:56.872 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:49:56.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4dc4fade-2c84-4875-add3-c42d38917fa2 2023-11-02T08:49:56.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4dc4fade-2c84-4875-add3-c42d38917fa2' 2023-11-02T08:49:56.873 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 4dc4fade-2c84-4875-add3-c42d38917fa2 2023-11-02T08:49:56.874 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:49:56.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC0YkNlnba4MxAAcMQjgvyUQR2wmIRr2ngYug== 2023-11-02T08:49:56.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC0YkNlnba4MxAAcMQjgvyUQR2wmIRr2ngYug=="}' 2023-11-02T08:49:56.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4dc4fade-2c84-4875-add3-c42d38917fa2 -i td/ec-error-rollforward/1/new.json 2023-11-02T08:49:57.388 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:49:57.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/1/new.json 2023-11-02T08:49:57.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0YkNlnba4MxAAcMQjgvyUQR2wmIRr2ngYug== --osd-uuid 4dc4fade-2c84-4875-add3-c42d38917fa2 2023-11-02T08:49:57.425 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:49:57.422+0000 7f9061c69500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:49:57.427 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:49:57.424+0000 7f9061c69500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:49:57.427 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:49:57.425+0000 7f9061c69500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:49:57.428 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:49:57.425+0000 7f9061c69500 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2023-11-02T08:49:57.428 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:49:57.425+0000 7f9061c69500 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2023-11-02T08:49:57.429 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:49:57.425+0000 7f9061c69500 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2023-11-02T08:49:57.504 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:49:57.502+0000 7f9061c69500 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2023-11-02T08:50:03.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2023-11-02T08:50:03.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:50:03.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-11-02T08:50:03.316 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2023-11-02T08:50:03.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:50:03.861 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T08:50:03.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-11-02T08:50:03.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:50:03.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:50:03.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:50:03.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:50:03.886 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:50:03.881+0000 7f7bd7410500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:50:03.910 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:50:03.905+0000 7f7bd7410500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:50:03.911 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:50:03.907+0000 7f7bd7410500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:50:04.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-11-02T08:50:04.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:50:04.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:50:04.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:50:04.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:50:04.274 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:50:04.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:04.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:50:04.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:04.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:50:04.447 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:50:04.444+0000 7f7bd7410500 -1 Falling back to public interface 2023-11-02T08:50:04.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:05.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:05.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:05.641 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:50:05.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:50:05.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:05.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:50:06.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:06.736 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:50:06.733+0000 7f7bd7410500 -1 osd.1 0 log_to_monitors true 2023-11-02T08:50:07.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:07.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:07.018 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:50:07.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:50:07.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:07.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:50:07.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:08.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:08.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:08.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:50:08.457 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:50:08.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:08.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:50:08.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:09.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:09.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:09.929 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:50:09.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:50:09.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:09.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:50:10.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:11.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:11.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:11.393 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:50:11.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:50:11.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:11.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:50:11.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:12.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:12.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:12.833 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:50:12.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:50:12.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:12.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:50:13.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:14.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:14.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:14.212 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:50:14.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:50:14.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:14.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:50:14.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:15.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:15.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:15.614 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:50:15.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:50:15.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:15.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:50:15.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:16.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:16.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:16.993 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:50:16.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:50:16.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:16.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:50:17.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:18.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:18.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:18.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:50:18.374 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:50:18.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:18.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:50:18.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:19.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:19.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:19.906 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:50:19.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:50:19.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:19.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:50:20.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:21.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:21.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:21.772 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:50:21.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:50:21.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:21.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:50:22.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:23.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:23.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:23.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:50:23.283 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:50:23.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:23.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:50:23.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:24.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:24.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:24.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:50:24.932 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:50:24.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:24.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:50:26.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:27.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:27.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:27.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:50:27.004 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:50:27.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:27.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:50:27.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:28.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:28.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:28.847 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:50:28.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:50:28.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:28.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:50:29.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:30.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:30.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:30.399 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T08:50:30.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T08:50:30.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:30.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:50:30.811 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/144613,v1:127.0.0.1:6811/144613] [v2:127.0.0.1:6812/144613,v1:127.0.0.1:6813/144613] exists,up 4dc4fade-2c84-4875-add3-c42d38917fa2 2023-11-02T08:50:30.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:50:30.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:50:30.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:50:30.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2023-11-02T08:50:30.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2023-11-02T08:50:30.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:50:30.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-11-02T08:50:30.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:50:30.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/2 2023-11-02T08:50:30.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2023-11-02T08:50:30.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:50:30.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:50:30.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:50:30.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2023-11-02T08:50:30.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2023-11-02T08:50:30.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:50:30.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:50:30.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2023-11-02T08:50:30.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:50:30.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:50:30.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:50:30.824 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:50:30.824 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:50:30.824 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141790 2023-11-02T08:50:30.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141790/$cluster-$name.asok' 2023-11-02T08:50:30.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' 2023-11-02T08:50:30.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:50:30.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:50:30.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:50:30.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2023-11-02T08:50:30.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2023-11-02T08:50:30.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:50:30.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:50:30.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:50:30.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:50:30.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:50:30.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:50:30.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/2 2023-11-02T08:50:30.827 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:50:30.828 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 342368a5-4b30-4a3c-908a-b3f829890a5b 2023-11-02T08:50:30.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=342368a5-4b30-4a3c-908a-b3f829890a5b 2023-11-02T08:50:30.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 342368a5-4b30-4a3c-908a-b3f829890a5b' 2023-11-02T08:50:30.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:50:30.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDWYkNlY9tLMRAAZJYwOg7QM0FvQ3nmyT7XTw== 2023-11-02T08:50:30.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDWYkNlY9tLMRAAZJYwOg7QM0FvQ3nmyT7XTw=="}' 2023-11-02T08:50:30.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 342368a5-4b30-4a3c-908a-b3f829890a5b -i td/ec-error-rollforward/2/new.json 2023-11-02T08:50:31.319 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:50:31.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/2/new.json 2023-11-02T08:50:31.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDWYkNlY9tLMRAAZJYwOg7QM0FvQ3nmyT7XTw== --osd-uuid 342368a5-4b30-4a3c-908a-b3f829890a5b 2023-11-02T08:50:31.355 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:50:31.352+0000 7f8f6f42a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:50:31.357 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:50:31.354+0000 7f8f6f42a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:50:31.357 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:50:31.355+0000 7f8f6f42a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:50:31.358 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:50:31.355+0000 7f8f6f42a500 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2023-11-02T08:50:31.358 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:50:31.355+0000 7f8f6f42a500 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2023-11-02T08:50:31.359 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:50:31.355+0000 7f8f6f42a500 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2023-11-02T08:50:31.426 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:50:31.423+0000 7f8f6f42a500 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2023-11-02T08:50:37.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2023-11-02T08:50:37.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:50:37.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-11-02T08:50:37.276 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2023-11-02T08:50:37.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:50:37.786 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T08:50:37.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-11-02T08:50:37.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:50:37.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:50:37.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:50:37.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:50:37.810 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:50:37.807+0000 7f3be2213500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:50:37.841 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:50:37.838+0000 7f3be2213500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:50:37.842 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:50:37.839+0000 7f3be2213500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:50:38.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-11-02T08:50:38.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:50:38.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T08:50:38.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:50:38.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:50:38.160 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:50:38.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:38.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:50:38.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:38.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:50:38.376 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:50:38.373+0000 7f3be2213500 -1 Falling back to public interface 2023-11-02T08:50:38.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:39.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:39.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:39.582 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:50:39.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:50:39.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:39.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:50:39.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:40.660 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:50:40.657+0000 7f3be2213500 -1 osd.2 0 log_to_monitors true 2023-11-02T08:50:40.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:40.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:40.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:50:40.958 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:50:40.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:40.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:50:41.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:42.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:42.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:42.455 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:50:42.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:50:42.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:42.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:50:42.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:43.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:43.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:43.837 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:50:43.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:50:43.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:43.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:50:44.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:45.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:45.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:45.399 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:50:45.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:50:45.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:45.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:50:45.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:46.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:46.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:46.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:50:46.813 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:50:46.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:46.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:50:47.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:48.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:48.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:48.219 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:50:48.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:50:48.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:48.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:50:48.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:49.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:49.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:49.642 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:50:49.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:50:49.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:49.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:50:50.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:51.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:51.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:51.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:50:51.023 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:50:51.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:51.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:50:51.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:52.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:52.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:52.433 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:50:52.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:50:52.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:52.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:50:52.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:53.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:53.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:53.860 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:50:53.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:50:53.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:53.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:50:54.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:55.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:55.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:55.448 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:50:55.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:50:55.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:55.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:50:55.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:56.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:56.991 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:50:56.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:56.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:50:56.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:56.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:50:57.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:50:58.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:50:58.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:50:58.782 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:50:58.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:50:58.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:50:58.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:50:59.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:51:00.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:51:00.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:00.403 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:51:00.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:51:00.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:00.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:51:01.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:51:02.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:51:02.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:02.386 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:51:02.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:51:02.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:02.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:51:03.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:51:04.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:51:04.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:04.171 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T08:51:04.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T08:51:04.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:04.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:51:04.664 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/146353,v1:127.0.0.1:6819/146353] [v2:127.0.0.1:6820/146353,v1:127.0.0.1:6821/146353] exists,up 342368a5-4b30-4a3c-908a-b3f829890a5b 2023-11-02T08:51:04.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:51:04.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:51:04.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:51:04.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2023-11-02T08:51:04.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2023-11-02T08:51:04.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:51:04.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2023-11-02T08:51:04.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:51:04.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/3 2023-11-02T08:51:04.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2023-11-02T08:51:04.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:51:04.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:51:04.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:51:04.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2023-11-02T08:51:04.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2023-11-02T08:51:04.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:51:04.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:51:04.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2023-11-02T08:51:04.672 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:51:04.672 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:51:04.673 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:51:04.673 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:51:04.673 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:51:04.674 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141790 2023-11-02T08:51:04.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141790/$cluster-$name.asok' 2023-11-02T08:51:04.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' 2023-11-02T08:51:04.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:51:04.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:51:04.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:51:04.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2023-11-02T08:51:04.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2023-11-02T08:51:04.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:51:04.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:51:04.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:51:04.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:51:04.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:51:04.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:51:04.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/3 2023-11-02T08:51:04.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:51:04.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5ccf5da5-7472-4a90-8ca2-197613d1e872 2023-11-02T08:51:04.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 5ccf5da5-7472-4a90-8ca2-197613d1e872' 2023-11-02T08:51:04.677 INFO:tasks.workunit.client.0.smithi116.stdout:add osd3 5ccf5da5-7472-4a90-8ca2-197613d1e872 2023-11-02T08:51:04.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:51:04.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD4YkNlj0qMKBAA0fLWhnt1cZxNrNkSMKqhAQ== 2023-11-02T08:51:04.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD4YkNlj0qMKBAA0fLWhnt1cZxNrNkSMKqhAQ=="}' 2023-11-02T08:51:04.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5ccf5da5-7472-4a90-8ca2-197613d1e872 -i td/ec-error-rollforward/3/new.json 2023-11-02T08:51:05.251 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:51:05.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/3/new.json 2023-11-02T08:51:05.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD4YkNlj0qMKBAA0fLWhnt1cZxNrNkSMKqhAQ== --osd-uuid 5ccf5da5-7472-4a90-8ca2-197613d1e872 2023-11-02T08:51:05.289 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:05.284+0000 7febfc42a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:05.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:05.305+0000 7febfc42a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:05.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:05.306+0000 7febfc42a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:05.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:05.306+0000 7febfc42a500 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2023-11-02T08:51:05.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:05.306+0000 7febfc42a500 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2023-11-02T08:51:05.311 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:05.306+0000 7febfc42a500 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2023-11-02T08:51:05.402 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:05.397+0000 7febfc42a500 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2023-11-02T08:51:11.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2023-11-02T08:51:11.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:51:11.173 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd3 key to auth repository 2023-11-02T08:51:11.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2023-11-02T08:51:11.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:51:11.695 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.3 2023-11-02T08:51:11.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2023-11-02T08:51:11.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:51:11.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:51:11.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:51:11.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:51:11.719 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:11.716+0000 7fb5222d8500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:11.752 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:11.749+0000 7fb5222d8500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:11.753 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:11.750+0000 7fb5222d8500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:12.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2023-11-02T08:51:12.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:51:12.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2023-11-02T08:51:12.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:51:12.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:51:12.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:12.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:51:12.074 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:51:12.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:12.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:51:12.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:12.276+0000 7fb5222d8500 -1 Falling back to public interface 2023-11-02T08:51:12.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:51:13.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:51:13.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:13.481 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:51:13.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:51:13.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:13.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:51:13.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:51:14.592 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:14.589+0000 7fb5222d8500 -1 osd.3 0 log_to_monitors true 2023-11-02T08:51:14.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:51:14.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:14.861 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:51:14.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:51:14.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:14.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:51:15.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:51:16.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:51:16.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:16.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:51:16.401 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:51:16.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:16.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:51:16.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:51:17.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:51:17.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:17.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:51:17.920 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:51:17.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:17.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:51:18.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:51:19.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:51:19.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:19.317 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:51:19.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:51:19.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:19.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:51:19.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:51:20.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:51:20.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:20.734 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:51:20.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:51:20.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:20.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:51:21.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:51:22.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:51:22.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:22.146 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:51:22.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:51:22.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:22.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:51:22.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:51:23.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:51:23.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:23.580 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:51:23.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:51:23.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:23.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:51:23.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:51:24.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:51:24.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:24.974 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:51:24.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:51:24.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:24.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:51:25.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:51:26.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:51:26.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:26.497 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:51:26.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:51:26.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:26.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:51:27.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:51:28.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:51:28.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:28.571 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:51:28.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:51:28.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:28.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:51:29.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:51:30.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:51:30.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:30.450 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:51:30.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:51:30.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:30.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:51:31.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:51:32.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:51:32.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:32.212 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:51:32.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:51:32.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:32.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:51:32.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:51:33.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:51:33.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:33.907 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:51:33.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:51:33.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:33.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:51:34.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:51:35.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:51:35.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:35.498 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:51:35.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:51:35.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:35.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:51:36.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:51:37.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:51:37.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:37.160 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:51:37.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:51:37.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:37.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:51:37.599 INFO:tasks.workunit.client.0.smithi116.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/148092,v1:127.0.0.1:6827/148092] [v2:127.0.0.1:6828/148092,v1:127.0.0.1:6829/148092] exists,up 5ccf5da5-7472-4a90-8ca2-197613d1e872 2023-11-02T08:51:37.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:51:37.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:51:37.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:51:37.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2023-11-02T08:51:38.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2023-11-02T08:51:38.892 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'ec' already exists 2023-11-02T08:51:38.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2023-11-02T08:51:40.185 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2023-11-02T08:51:40.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 146353 2023-11-02T08:51:40.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=149039 2023-11-02T08:51:40.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2023-11-02T08:51:40.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2023-11-02T08:51:41.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 149060' 2023-11-02T08:51:41.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2023-11-02T08:51:41.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 149061' 2023-11-02T08:51:41.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2023-11-02T08:51:41.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 149062' 2023-11-02T08:51:41.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2023-11-02T08:51:41.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2023-11-02T08:51:42.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2023-11-02T08:51:42.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:51:42.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:51:42.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:51:42.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:51:42.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:51:42.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:51:42.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 149039 149060 149061 149062 2023-11-02T08:51:42.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2023-11-02T08:51:42.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2023-11-02T08:51:42.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2023-11-02T08:51:42.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:51:42.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-11-02T08:51:42.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:51:42.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/0 2023-11-02T08:51:42.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2023-11-02T08:51:42.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:51:42.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:51:42.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:51:42.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2023-11-02T08:51:42.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2023-11-02T08:51:42.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:51:42.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:51:42.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2023-11-02T08:51:42.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:51:42.308 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:51:42.308 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:51:42.308 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:51:42.308 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:51:42.308 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141790 2023-11-02T08:51:42.308 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141790/$cluster-$name.asok' 2023-11-02T08:51:42.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' 2023-11-02T08:51:42.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:51:42.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2023-11-02T08:51:42.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2023-11-02T08:51:42.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:51:42.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:51:42.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:51:42.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:51:42.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:51:42.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:51:42.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/0 2023-11-02T08:51:42.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-11-02T08:51:42.311 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T08:51:42.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:51:42.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/0/whoami 2023-11-02T08:51:42.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-11-02T08:51:42.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:51:42.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:51:42.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:51:42.329 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:42.326+0000 7f7c71a14500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:42.331 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:42.328+0000 7f7c71a14500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:42.332 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:42.329+0000 7f7c71a14500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:42.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-11-02T08:51:42.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:51:42.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:51:42.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:51:42.718 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:51:42.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:51:42.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:42.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:51:42.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:42.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:51:42.869 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:42.866+0000 7f7c71a14500 -1 Falling back to public interface 2023-11-02T08:51:43.080 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/142874,v1:127.0.0.1:6803/142874] [v2:127.0.0.1:6804/142874,v1:127.0.0.1:6805/142874] exists,up aaa308f4-98eb-42a0-9e17-452d97fb0f43 2023-11-02T08:51:43.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:51:43.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:51:43.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:51:43.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2023-11-02T08:51:43.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2023-11-02T08:51:43.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:51:43.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2023-11-02T08:51:43.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:51:43.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/1 2023-11-02T08:51:43.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2023-11-02T08:51:43.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:51:43.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:51:43.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:51:43.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2023-11-02T08:51:43.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2023-11-02T08:51:43.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:51:43.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:51:43.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2023-11-02T08:51:43.088 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:51:43.088 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:51:43.088 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:51:43.088 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:51:43.088 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:51:43.089 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141790 2023-11-02T08:51:43.089 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141790/$cluster-$name.asok' 2023-11-02T08:51:43.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' 2023-11-02T08:51:43.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:51:43.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2023-11-02T08:51:43.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2023-11-02T08:51:43.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:51:43.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:51:43.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:51:43.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:51:43.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:51:43.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:51:43.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/1 2023-11-02T08:51:43.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2023-11-02T08:51:43.091 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T08:51:43.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:51:43.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/1/whoami 2023-11-02T08:51:43.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2023-11-02T08:51:43.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:51:43.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:51:43.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:51:43.107 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:43.104+0000 7f53c42b6500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:43.117 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:43.114+0000 7f53c42b6500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:43.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:43.115+0000 7f53c42b6500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:43.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2023-11-02T08:51:43.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:51:43.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:51:43.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:51:43.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:51:43.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:43.498 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:51:43.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:51:43.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:43.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:51:43.637 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:43.634+0000 7f53c42b6500 -1 Falling back to public interface 2023-11-02T08:51:43.872 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 23 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/144613,v1:127.0.0.1:6811/144613] [v2:127.0.0.1:6812/144613,v1:127.0.0.1:6813/144613] exists,up 4dc4fade-2c84-4875-add3-c42d38917fa2 2023-11-02T08:51:43.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:51:43.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:51:43.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:51:43.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2023-11-02T08:51:43.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2023-11-02T08:51:43.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:51:43.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2023-11-02T08:51:43.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:51:43.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/2 2023-11-02T08:51:43.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2023-11-02T08:51:43.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:51:43.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:51:43.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:51:43.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2023-11-02T08:51:43.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2023-11-02T08:51:43.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:51:43.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:51:43.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2023-11-02T08:51:43.880 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:51:43.880 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:51:43.880 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:51:43.881 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:51:43.881 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:51:43.882 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141790 2023-11-02T08:51:43.882 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141790/$cluster-$name.asok' 2023-11-02T08:51:43.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' 2023-11-02T08:51:43.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:51:43.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2023-11-02T08:51:43.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2023-11-02T08:51:43.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:51:43.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:51:43.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:51:43.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:51:43.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:51:43.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:51:43.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/2 2023-11-02T08:51:43.884 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T08:51:43.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2023-11-02T08:51:43.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:51:43.886 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/2/whoami 2023-11-02T08:51:43.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2023-11-02T08:51:43.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:51:43.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:51:43.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:51:43.897 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:43.894+0000 7f4b92fad500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:43.920 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:43.917+0000 7f4b92fad500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:43.921 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:43.918+0000 7f4b92fad500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:44.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2023-11-02T08:51:44.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:51:44.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T08:51:44.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:51:44.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:51:44.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:44.292 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:51:44.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:51:44.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:44.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:51:44.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:44.439+0000 7f4b92fad500 -1 Falling back to public interface 2023-11-02T08:51:44.699 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/146353,v1:127.0.0.1:6819/146353] [v2:127.0.0.1:6820/146353,v1:127.0.0.1:6821/146353] exists,up 342368a5-4b30-4a3c-908a-b3f829890a5b 2023-11-02T08:51:44.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:51:44.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:51:44.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:51:44.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2023-11-02T08:51:44.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2023-11-02T08:51:44.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:51:44.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2023-11-02T08:51:44.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:51:44.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/3 2023-11-02T08:51:44.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2023-11-02T08:51:44.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:51:44.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:51:44.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:51:44.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2023-11-02T08:51:44.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2023-11-02T08:51:44.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:51:44.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:51:44.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2023-11-02T08:51:44.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:51:44.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:51:44.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:51:44.707 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:51:44.708 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:51:44.708 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141790 2023-11-02T08:51:44.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.141790/$cluster-$name.asok' 2023-11-02T08:51:44.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' 2023-11-02T08:51:44.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:51:44.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2023-11-02T08:51:44.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2023-11-02T08:51:44.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:51:44.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:51:44.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:51:44.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:51:44.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:51:44.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:51:44.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/3 2023-11-02T08:51:44.711 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.3 2023-11-02T08:51:44.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2023-11-02T08:51:44.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=36c1cf20-a55a-4ce0-b3f7-a3211113fdeb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.141790/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:51:44.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/3/whoami 2023-11-02T08:51:44.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2023-11-02T08:51:44.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:51:44.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:51:44.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:51:44.725 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:44.722+0000 7f1a4d80f500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:44.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:44.724+0000 7f1a4d80f500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:44.728 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:44.725+0000 7f1a4d80f500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:45.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2023-11-02T08:51:45.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:51:45.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2023-11-02T08:51:45.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:51:45.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:51:45.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:51:45.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:51:45.095 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:51:45.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:51:45.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:51:45.253 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:45.250+0000 7f1a4d80f500 -1 Falling back to public interface 2023-11-02T08:51:45.547 INFO:tasks.workunit.client.0.smithi116.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/148092,v1:127.0.0.1:6827/148092] [v2:127.0.0.1:6828/148092,v1:127.0.0.1:6829/148092] exists,up 5ccf5da5-7472-4a90-8ca2-197613d1e872 2023-11-02T08:51:45.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:51:45.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:51:45.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:51:45.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2023-11-02T08:51:45.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T08:51:45.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T08:51:45.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T08:51:45.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-11-02T08:51:45.554 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T08:51:45.554 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T08:51:45.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T08:51:45.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T08:51:45.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T08:51:45.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-11-02T08:51:45.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T08:51:45.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T08:51:45.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T08:51:45.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:51:45.629 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:51:46.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:51:46.026 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:51:46.026 INFO:tasks.workunit.client.0.smithi116.stderr:2 2023-11-02T08:51:46.026 INFO:tasks.workunit.client.0.smithi116.stderr:3' 2023-11-02T08:51:46.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:51:46.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:51:46.027 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:51:46.214 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:46.209+0000 7f7c71a14500 -1 osd.0 25 log_to_monitors true 2023-11-02T08:51:47.018 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:47.015+0000 7f53c42b6500 -1 osd.1 25 log_to_monitors true 2023-11-02T08:51:48.044 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:48.041+0000 7f4b92fad500 -1 osd.2 24 log_to_monitors true 2023-11-02T08:51:48.882 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:48.879+0000 7f1a4d80f500 -1 osd.3 25 log_to_monitors true 2023-11-02T08:51:51.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=120259084291 2023-11-02T08:51:51.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 120259084291 2023-11-02T08:51:51.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-120259084291' 2023-11-02T08:51:51.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:51:51.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:51:51.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=133143986179 2023-11-02T08:51:51.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 133143986179 2023-11-02T08:51:51.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-120259084291 1-133143986179' 2023-11-02T08:51:51.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:51:51.361 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:51:51.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=141733920771 2023-11-02T08:51:51.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 141733920771 2023-11-02T08:51:51.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-120259084291 1-133143986179 2-141733920771' 2023-11-02T08:51:51.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:51:51.470 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-11-02T08:51:51.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=150323855363 2023-11-02T08:51:51.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 150323855363 2023-11-02T08:51:51.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-120259084291 1-133143986179 2-141733920771 3-150323855363' 2023-11-02T08:51:51.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:51:51.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-120259084291 2023-11-02T08:51:51.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:51:51.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:51:51.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-120259084291 2023-11-02T08:51:51.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:51:51.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=120259084291 2023-11-02T08:51:51.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 120259084291' 2023-11-02T08:51:51.582 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 120259084291 2023-11-02T08:51:51.583 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:51:51.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 120259084292 -lt 120259084291 2023-11-02T08:51:51.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:51:51.976 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-133143986179 2023-11-02T08:51:51.976 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:51:51.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T08:51:51.977 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-133143986179 2023-11-02T08:51:51.977 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:51:51.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=133143986179 2023-11-02T08:51:51.978 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 133143986179 2023-11-02T08:51:51.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 133143986179' 2023-11-02T08:51:51.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T08:51:52.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 133143986179 -lt 133143986179 2023-11-02T08:51:52.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:51:52.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-141733920771 2023-11-02T08:51:52.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:51:52.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:51:52.349 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-141733920771 2023-11-02T08:51:52.349 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:51:52.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=141733920771 2023-11-02T08:51:52.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 141733920771' 2023-11-02T08:51:52.351 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 141733920771 2023-11-02T08:51:52.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:51:52.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 141733920771 -lt 141733920771 2023-11-02T08:51:52.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:51:52.753 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-150323855363 2023-11-02T08:51:52.753 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:51:52.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2023-11-02T08:51:52.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-150323855363 2023-11-02T08:51:52.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:51:52.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=150323855363 2023-11-02T08:51:52.756 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 150323855363 2023-11-02T08:51:52.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 150323855363' 2023-11-02T08:51:52.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2023-11-02T08:51:53.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 150323855363 -lt 150323855363 2023-11-02T08:51:53.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T08:51:53.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:51:53.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:51:53.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2023-11-02T08:51:53.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:51:53.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:51:53.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:51:53.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:51:53.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:51:53.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:51:53.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:51:54.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T08:51:54.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:51:54.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:51:54.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:51:54.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2023-11-02T08:51:54.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T08:51:54.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T08:51:54.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2023-11-02T08:51:54.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2023-11-02T08:51:54.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T08:51:54.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2023-11-02T08:51:54.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:51:54.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:51:54.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:51:54.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:51:54.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:51:54.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:51:54.657 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T08:51:54.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T08:51:54.657 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T08:51:54.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T08:51:54.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T08:51:54.659 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T08:51:54.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:51:54.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T08:51:54.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T08:51:54.660 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:51:54.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T08:51:54.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T08:51:54.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2023-11-02T08:51:54.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T08:51:54.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:51:54.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141790 2023-11-02T08:51:54.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.141790 2023-11-02T08:51:54.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T08:51:54.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T08:51:54.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2023-11-02T08:51:54.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/ec-error-rollforward 0 2023-11-02T08:51:54.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2023-11-02T08:51:54.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-11-02T08:51:54.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2023-11-02T08:51:54.682 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:51:54.682 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:51:54.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:51:54.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:51:54.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:51:54.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:51:54.685 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T08:51:54.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T08:51:54.685 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T08:51:54.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T08:51:54.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T08:51:54.687 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T08:51:54.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:51:54.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T08:51:54.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T08:51:54.688 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:51:54.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T08:51:54.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-11-02T08:51:54.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2023-11-02T08:51:54.690 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T08:51:54.691 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:51:54.691 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.141790 2023-11-02T08:51:54.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.141790 2023-11-02T08:51:54.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T08:51:54.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T08:51:54.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2023-11-02T08:51:54.693 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-02T08:51:54.694 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-02T08:51:54.758 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2023-11-02T08:51:54.758 DEBUG:teuthology.orchestra.run.smithi116:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2023-11-02T08:51:54.818 INFO:tasks.workunit.client.0.smithi116.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-11-02T08:51:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-11-02T08:51:54.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T08:51:54.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T08:51:54.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2023-11-02T08:51:54.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2023-11-02T08:51:54.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2023-11-02T08:51:54.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2023-11-02T08:51:54.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2023-11-02T08:51:54.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2023-11-02T08:51:54.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-bench 2023-11-02T08:51:54.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2023-11-02T08:51:54.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2023-11-02T08:51:54.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2023-11-02T08:51:54.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2023-11-02T08:51:54.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2023-11-02T08:51:54.827 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2023-11-02T08:51:54.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=2787d054-23fb-4ab2-b2b2-dd56c6cfe8fc --auth-supported=none ' 2023-11-02T08:51:54.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2023-11-02T08:51:54.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2023-11-02T08:51:54.828 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2023-11-02T08:51:54.828 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-11-02T08:51:54.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2023-11-02T08:51:54.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2023-11-02T08:51:54.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2023-11-02T08:51:54.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bench 2023-11-02T08:51:54.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bench 2023-11-02T08:51:54.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2023-11-02T08:51:54.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T08:51:54.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2023-11-02T08:51:54.832 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:51:54.832 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:51:54.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:51:54.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:51:54.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:51:54.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:51:54.833 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T08:51:54.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T08:51:54.834 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T08:51:54.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T08:51:54.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T08:51:54.834 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T08:51:54.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:51:54.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T08:51:54.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T08:51:54.835 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:51:54.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T08:51:54.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T08:51:54.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2023-11-02T08:51:54.838 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T08:51:54.839 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:51:54.841 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151572 2023-11-02T08:51:54.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.151572 2023-11-02T08:51:54.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T08:51:54.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T08:51:54.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bench 2023-11-02T08:51:54.843 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T08:51:54.843 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:51:54.843 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151572 2023-11-02T08:51:54.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.151572 2023-11-02T08:51:54.844 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T08:51:54.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-11-02T08:51:54.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-11-02T08:51:54.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T08:51:54.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2023-11-02T08:51:54.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2023-11-02T08:51:54.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2023-11-02T08:51:54.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2023-11-02T08:51:54.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bench 2023-11-02T08:51:54.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T08:51:54.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T08:51:54.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T08:51:54.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bench/a 2023-11-02T08:51:54.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2023-11-02T08:51:55.157 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T08:51:55.157 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:51:55.157 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:51:55.158 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:51:55.158 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:51:55.158 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151572 2023-11-02T08:51:55.158 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.151572/$cluster-$name.asok' 2023-11-02T08:51:55.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.151572/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T08:51:55.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T08:51:55.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T08:51:55.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T08:51:55.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T08:51:55.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T08:51:55.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T08:51:55.341 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T08:51:55.341 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T08:51:55.341 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T08:51:55.342 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:51:55.342 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:51:55.342 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151572 2023-11-02T08:51:55.343 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.151572/ceph-mon.a.asok 2023-11-02T08:51:55.343 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T08:51:55.343 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.151572/ceph-mon.a.asok config get fsid 2023-11-02T08:51:55.418 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T08:51:55.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T08:51:55.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T08:51:55.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T08:51:55.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T08:51:55.419 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T08:51:55.420 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T08:51:55.420 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T08:51:55.420 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:51:55.420 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:51:55.420 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151572 2023-11-02T08:51:55.421 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.151572/ceph-mon.a.asok 2023-11-02T08:51:55.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T08:51:55.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.151572/ceph-mon.a.asok config get mon_host 2023-11-02T08:51:55.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2023-11-02T08:51:55.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bench 2023-11-02T08:51:55.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T08:51:55.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T08:51:55.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T08:51:55.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bench/x 2023-11-02T08:51:55.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T08:51:56.343 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T08:51:56.344 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:51:56.344 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:51:56.344 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:51:56.345 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:51:56.345 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151572 2023-11-02T08:51:56.345 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.151572/$cluster-$name.asok' 2023-11-02T08:51:56.345 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T08:51:56.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.151572/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T08:51:56.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2023-11-02T08:51:56.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bench 2023-11-02T08:51:56.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:51:56.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T08:51:56.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:51:56.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bench/0 2023-11-02T08:51:56.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2787d054-23fb-4ab2-b2b2-dd56c6cfe8fc --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2023-11-02T08:51:56.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:51:56.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:51:56.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:51:56.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2023-11-02T08:51:56.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2023-11-02T08:51:56.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:51:56.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:51:56.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2023-11-02T08:51:56.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:51:56.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:51:56.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:51:56.377 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:51:56.377 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:51:56.378 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151572 2023-11-02T08:51:56.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.151572/$cluster-$name.asok' 2023-11-02T08:51:56.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.151572/$cluster-$name.asok' 2023-11-02T08:51:56.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:51:56.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:51:56.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:51:56.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2023-11-02T08:51:56.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2023-11-02T08:51:56.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:51:56.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:51:56.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:51:56.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:51:56.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:51:56.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:51:56.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bench/0 2023-11-02T08:51:56.382 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:51:56.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c8039c7b-972d-4da5-8361-267c65feaa94 2023-11-02T08:51:56.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c8039c7b-972d-4da5-8361-267c65feaa94' 2023-11-02T08:51:56.383 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 c8039c7b-972d-4da5-8361-267c65feaa94 2023-11-02T08:51:56.383 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:51:56.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAsY0NllGL/FhAATbOYHvgdl/UQORUa7ZAn2w== 2023-11-02T08:51:56.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAsY0NllGL/FhAATbOYHvgdl/UQORUa7ZAn2w=="}' 2023-11-02T08:51:56.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c8039c7b-972d-4da5-8361-267c65feaa94 -i td/osd-bench/0/new.json 2023-11-02T08:51:56.731 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:51:56.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bench/0/new.json 2023-11-02T08:51:56.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2787d054-23fb-4ab2-b2b2-dd56c6cfe8fc --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.151572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAsY0NllGL/FhAATbOYHvgdl/UQORUa7ZAn2w== --osd-uuid c8039c7b-972d-4da5-8361-267c65feaa94 2023-11-02T08:51:56.767 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:56.764+0000 7fc38f974500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:56.769 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:56.766+0000 7fc38f974500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:56.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:56.767+0000 7fc38f974500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:51:56.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:56.767+0000 7fc38f974500 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2023-11-02T08:51:56.770 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:56.767+0000 7fc38f974500 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2023-11-02T08:51:56.771 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:56.767+0000 7fc38f974500 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2023-11-02T08:51:56.821 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:51:56.818+0000 7fc38f974500 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2023-11-02T08:52:02.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bench/0/keyring 2023-11-02T08:52:02.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:52:02.178 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T08:52:02.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T08:52:02.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:52:02.714 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T08:52:02.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T08:52:02.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2787d054-23fb-4ab2-b2b2-dd56c6cfe8fc --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.151572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:52:02.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:52:02.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:52:02.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:52:02.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:52:02.736+0000 7f683ba0d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:52:02.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:52:02.738+0000 7f683ba0d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:52:02.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:52:02.739+0000 7f683ba0d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:52:03.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T08:52:03.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:52:03.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:52:03.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:52:03.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:52:03.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:52:03.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:52:03.168 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:52:03.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:52:03.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:52:03.265 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:52:03.262+0000 7f683ba0d500 -1 Falling back to public interface 2023-11-02T08:52:03.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:52:04.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:52:04.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:52:04.577 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:52:04.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:52:04.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:52:04.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:52:04.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:52:05.595 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:52:05.592+0000 7f683ba0d500 -1 osd.0 0 log_to_monitors true 2023-11-02T08:52:05.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:52:05.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:52:05.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:52:05.995 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:52:05.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:52:05.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:52:06.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:52:07.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:52:07.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:52:07.370 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:52:07.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:52:07.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:52:07.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:52:07.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:52:08.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:52:08.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:52:08.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:52:08.779 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:52:08.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:52:08.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:52:09.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:52:10.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:52:10.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:52:10.157 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:52:10.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:52:10.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:52:10.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:52:10.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:52:11.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:52:11.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:52:11.657 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:52:11.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:52:11.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:52:11.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:52:12.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:52:13.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:52:13.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:52:13.072 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:52:13.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:52:13.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:52:13.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:52:13.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:52:14.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:52:14.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:52:14.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:52:14.454 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:52:14.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:52:14.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:52:14.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:52:15.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:52:15.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:52:15.849 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:52:15.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:52:15.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:52:15.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:52:16.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:52:17.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:52:17.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:52:17.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:52:17.340 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:52:17.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:52:17.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:52:18.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:52:19.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:52:19.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:52:19.119 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:52:19.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:52:19.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:52:19.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:52:19.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:52:20.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:52:20.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:52:20.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:52:20.843 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:52:20.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:52:20.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:52:22.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:52:23.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:52:23.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:52:23.174 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:52:23.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:52:23.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:52:23.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:52:23.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:52:24.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:52:24.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:52:24.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:52:24.754 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:52:24.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:52:24.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:52:25.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:52:26.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:52:26.508 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:52:26.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:52:26.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:52:26.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:52:26.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:52:27.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:52:28.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:52:28.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:52:28.531 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:52:28.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:52:28.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:52:28.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:52:28.921 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/152656,v1:127.0.0.1:6803/152656] [v2:127.0.0.1:6804/152656,v1:127.0.0.1:6805/152656] exists,up c8039c7b-972d-4da5-8361-267c65feaa94 2023-11-02T08:52:28.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:52:28.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:52:28.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:52:28.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2023-11-02T08:52:28.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2023-11-02T08:52:28.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2023-11-02T08:52:28.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2023-11-02T08:52:28.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2023-11-02T08:52:28.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2023-11-02T08:52:28.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2023-11-02T08:52:28.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2023-11-02T08:52:28.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2023-11-02T08:52:28.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2023-11-02T08:52:28.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2023-11-02T08:52:28.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2023-11-02T08:52:28.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2023-11-02T08:52:28.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: expect_failure: local dir=td/osd-bench 2023-11-02T08:52:28.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: expect_failure: shift 2023-11-02T08:52:28.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: expect_failure: local expected=osd_bench_max_block_size 2023-11-02T08:52:28.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: expect_failure: shift 2023-11-02T08:52:28.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1921: expect_failure: local success 2023-11-02T08:52:28.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: expect_failure: ceph tell osd.0 bench 1024 67108865 2023-11-02T08:52:29.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: expect_failure: success=false 2023-11-02T08:52:29.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: false 2023-11-02T08:52:29.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2023-11-02T08:52:29.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: return 0 2023-11-02T08:52:29.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2023-11-02T08:52:29.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2023-11-02T08:52:29.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2023-11-02T08:52:29.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: expect_failure: local dir=td/osd-bench 2023-11-02T08:52:29.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: expect_failure: shift 2023-11-02T08:52:29.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: expect_failure: local expected=bench_small_size_max_iops 2023-11-02T08:52:29.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: expect_failure: shift 2023-11-02T08:52:29.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1921: expect_failure: local success 2023-11-02T08:52:29.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: expect_failure: ceph tell osd.0 bench 3072001 1024 2023-11-02T08:52:29.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: expect_failure: success=false 2023-11-02T08:52:29.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: false 2023-11-02T08:52:29.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2023-11-02T08:52:29.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: return 0 2023-11-02T08:52:29.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2023-11-02T08:52:29.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2023-11-02T08:52:29.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2023-11-02T08:52:29.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: expect_failure: local dir=td/osd-bench 2023-11-02T08:52:29.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: expect_failure: shift 2023-11-02T08:52:29.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: expect_failure: local expected=osd_bench_large_size_max_throughput 2023-11-02T08:52:29.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: expect_failure: shift 2023-11-02T08:52:29.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1921: expect_failure: local success 2023-11-02T08:52:29.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2023-11-02T08:52:29.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: expect_failure: success=false 2023-11-02T08:52:29.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: false 2023-11-02T08:52:29.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2023-11-02T08:52:29.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: return 0 2023-11-02T08:52:29.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2023-11-02T08:53:16.426 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T08:53:16.427 INFO:tasks.workunit.client.0.smithi116.stdout: "bytes_written": 1073741824, 2023-11-02T08:53:16.427 INFO:tasks.workunit.client.0.smithi116.stdout: "blocksize": 4194304, 2023-11-02T08:53:16.427 INFO:tasks.workunit.client.0.smithi116.stdout: "elapsed_sec": 22.438003248000001, 2023-11-02T08:53:16.427 INFO:tasks.workunit.client.0.smithi116.stdout: "bytes_per_sec": 47853715.508117124, 2023-11-02T08:53:16.427 INFO:tasks.workunit.client.0.smithi116.stdout: "iops": 11.409214856175691 2023-11-02T08:53:16.428 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T08:53:16.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2023-11-02T08:53:16.861 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T08:53:16.861 INFO:tasks.workunit.client.0.smithi116.stdout: "bytes_written": 10, 2023-11-02T08:53:16.862 INFO:tasks.workunit.client.0.smithi116.stdout: "blocksize": 14456, 2023-11-02T08:53:16.862 INFO:tasks.workunit.client.0.smithi116.stdout: "elapsed_sec": 0.028088888999999999, 2023-11-02T08:53:16.862 INFO:tasks.workunit.client.0.smithi116.stdout: "bytes_per_sec": 356.01265681957022, 2023-11-02T08:53:16.862 INFO:tasks.workunit.client.0.smithi116.stdout: "iops": 0.024627328224928764 2023-11-02T08:53:16.863 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T08:53:16.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2023-11-02T08:53:17.097 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T08:53:17.098 INFO:tasks.workunit.client.0.smithi116.stdout: "bytes_written": 1, 2023-11-02T08:53:17.098 INFO:tasks.workunit.client.0.smithi116.stdout: "blocksize": 14456, 2023-11-02T08:53:17.098 INFO:tasks.workunit.client.0.smithi116.stdout: "elapsed_sec": 0.034353631000000003, 2023-11-02T08:53:17.098 INFO:tasks.workunit.client.0.smithi116.stdout: "bytes_per_sec": 29.109004518328788, 2023-11-02T08:53:17.099 INFO:tasks.workunit.client.0.smithi116.stdout: "iops": 0.0020136278720482006 2023-11-02T08:53:17.099 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T08:53:17.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2023-11-02T08:53:17.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2023-11-02T08:53:17.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T08:53:17.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2023-11-02T08:53:17.104 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:53:17.104 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:53:17.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:53:17.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:53:17.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:53:17.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:53:17.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T08:53:17.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T08:53:17.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T08:53:17.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T08:53:17.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T08:53:17.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T08:53:17.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:53:17.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T08:53:17.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T08:53:17.211 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:53:17.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T08:53:17.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T08:53:17.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2023-11-02T08:53:17.236 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T08:53:17.236 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:53:17.236 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151572 2023-11-02T08:53:17.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.151572 2023-11-02T08:53:17.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T08:53:17.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T08:53:17.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2023-11-02T08:53:17.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-bench 0 2023-11-02T08:53:17.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2023-11-02T08:53:17.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-11-02T08:53:17.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2023-11-02T08:53:17.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:53:17.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:53:17.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:53:17.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:53:17.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:53:17.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:53:17.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T08:53:17.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T08:53:17.256 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T08:53:17.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T08:53:17.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T08:53:17.256 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T08:53:17.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:53:17.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T08:53:17.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T08:53:17.258 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:53:17.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T08:53:17.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-11-02T08:53:17.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2023-11-02T08:53:17.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T08:53:17.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:53:17.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.151572 2023-11-02T08:53:17.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.151572 2023-11-02T08:53:17.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T08:53:17.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T08:53:17.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2023-11-02T08:53:17.261 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-02T08:53:17.261 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-02T08:53:17.313 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2023-11-02T08:53:17.314 DEBUG:teuthology.orchestra.run.smithi116:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2023-11-02T08:53:17.378 INFO:tasks.workunit.client.0.smithi116.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-11-02T08:53:17.383 INFO:tasks.workunit.client.0.smithi116.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-11-02T08:53:17.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T08:53:17.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T08:53:17.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2023-11-02T08:53:17.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2023-11-02T08:53:17.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2023-11-02T08:53:17.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2023-11-02T08:53:17.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2023-11-02T08:53:17.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2023-11-02T08:53:17.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-bluefs-volume-ops 2023-11-02T08:53:17.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2023-11-02T08:53:17.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2023-11-02T08:53:17.385 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: set 2023-11-02T08:53:17.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-11-02T08:53:17.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: local 'funcs=TEST_bluestore 2023-11-02T08:53:17.389 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_bluestore2 2023-11-02T08:53:17.389 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_bluestore_expand' 2023-11-02T08:53:17.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2023-11-02T08:53:17.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2023-11-02T08:53:17.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2023-11-02T08:53:17.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2023-11-02T08:53:17.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2023-11-02T08:53:17.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T08:53:17.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2023-11-02T08:53:17.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T08:53:17.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T08:53:17.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T08:53:17.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T08:53:17.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T08:53:17.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T08:53:17.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T08:53:17.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T08:53:17.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T08:53:17.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T08:53:17.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T08:53:17.393 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T08:53:17.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:53:17.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T08:53:17.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T08:53:17.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T08:53:17.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T08:53:17.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T08:53:17.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2023-11-02T08:53:17.396 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T08:53:17.396 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:53:17.397 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T08:53:17.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153754 2023-11-02T08:53:17.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T08:53:17.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T08:53:17.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2023-11-02T08:53:17.399 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T08:53:17.399 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:53:17.399 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T08:53:17.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.153754 2023-11-02T08:53:17.403 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T08:53:17.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-11-02T08:53:17.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-11-02T08:53:17.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T08:53:17.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2023-11-02T08:53:17.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore td/osd-bluefs-volume-ops 2023-11-02T08:53:17.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2023-11-02T08:53:17.404 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: ulimit -n 2023-11-02T08:53:17.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: local flimit=4096 2023-11-02T08:53:17.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: TEST_bluestore: '[' 4096 -lt 1536 ']' 2023-11-02T08:53:17.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: export CEPH_MON=127.0.0.1:7146 2023-11-02T08:53:17.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: CEPH_MON=127.0.0.1:7146 2023-11-02T08:53:17.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:27: TEST_bluestore: export CEPH_ARGS 2023-11-02T08:53:17.406 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: uuidgen 2023-11-02T08:53:17.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: CEPH_ARGS+='--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none ' 2023-11-02T08:53:17.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:29: TEST_bluestore: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2023-11-02T08:53:17.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2023-11-02T08:53:17.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:31: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_create=true ' 2023-11-02T08:53:17.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2023-11-02T08:53:17.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2023-11-02T08:53:17.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:34: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2023-11-02T08:53:17.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:35: TEST_bluestore: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2023-11-02T08:53:17.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2023-11-02T08:53:17.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2023-11-02T08:53:17.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T08:53:17.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T08:53:17.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T08:53:17.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2023-11-02T08:53:17.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2023-11-02T08:53:17.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T08:53:17.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:53:17.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:53:17.776 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:53:17.776 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:53:17.776 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T08:53:17.776 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:53:17.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T08:53:17.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T08:53:17.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T08:53:17.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T08:53:17.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T08:53:17.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T08:53:17.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T08:53:17.962 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T08:53:17.962 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T08:53:17.962 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T08:53:17.963 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:53:17.963 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:53:17.963 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T08:53:17.964 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153754/ceph-mon.a.asok 2023-11-02T08:53:17.964 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T08:53:17.964 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.153754/ceph-mon.a.asok config get fsid 2023-11-02T08:53:18.040 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T08:53:18.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T08:53:18.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T08:53:18.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T08:53:18.042 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T08:53:18.042 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T08:53:18.042 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T08:53:18.042 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T08:53:18.042 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T08:53:18.043 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:53:18.043 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T08:53:18.043 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153754/ceph-mon.a.asok 2023-11-02T08:53:18.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T08:53:18.044 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.153754/ceph-mon.a.asok config get mon_host 2023-11-02T08:53:18.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2023-11-02T08:53:18.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2023-11-02T08:53:18.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T08:53:18.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T08:53:18.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T08:53:18.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2023-11-02T08:53:18.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T08:53:18.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T08:53:18.958 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:53:18.958 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:53:18.958 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:53:18.959 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:53:18.959 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T08:53:18.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:53:18.960 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T08:53:18.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T08:53:18.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2023-11-02T08:53:18.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2023-11-02T08:53:18.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:53:18.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T08:53:18.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:53:18.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2023-11-02T08:53:18.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2023-11-02T08:53:18.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:53:18.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:53:18.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:53:18.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2023-11-02T08:53:18.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2023-11-02T08:53:18.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:53:18.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:53:18.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2023-11-02T08:53:18.994 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:53:18.994 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:53:18.995 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:53:18.995 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:53:18.995 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:53:18.996 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T08:53:18.996 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:53:18.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:53:18.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:53:18.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:53:18.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:53:18.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2023-11-02T08:53:18.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2023-11-02T08:53:18.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:53:19.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:53:19.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:53:19.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:53:19.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:53:19.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:53:19.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2023-11-02T08:53:19.001 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:53:19.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b30aad1c-508b-4a9d-871f-a21c604cf0a2 2023-11-02T08:53:19.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b30aad1c-508b-4a9d-871f-a21c604cf0a2' 2023-11-02T08:53:19.002 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 b30aad1c-508b-4a9d-871f-a21c604cf0a2 2023-11-02T08:53:19.002 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:53:19.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB+Y0NlVK2LOxAA8r176YDPgsJznjcXaXTSBg== 2023-11-02T08:53:19.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB+Y0NlVK2LOxAA8r176YDPgsJznjcXaXTSBg=="}' 2023-11-02T08:53:19.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b30aad1c-508b-4a9d-871f-a21c604cf0a2 -i td/osd-bluefs-volume-ops/0/new.json 2023-11-02T08:53:19.310 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:53:19.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2023-11-02T08:53:19.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB+Y0NlVK2LOxAA8r176YDPgsJznjcXaXTSBg== --osd-uuid b30aad1c-508b-4a9d-871f-a21c604cf0a2 2023-11-02T08:53:19.348 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:53:19.343+0000 7f1a0e8c0500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:53:19.349 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:53:19.345+0000 7f1a0e8c0500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:53:19.350 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:53:19.346+0000 7f1a0e8c0500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:53:19.351 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:53:19.346+0000 7f1a0e8c0500 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2023-11-02T08:53:19.351 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:53:19.346+0000 7f1a0e8c0500 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2023-11-02T08:53:19.351 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:53:19.346+0000 7f1a0e8c0500 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2023-11-02T08:53:19.426 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:53:19.422+0000 7f1a0e8c0500 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2023-11-02T08:53:28.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2023-11-02T08:53:28.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:53:28.133 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T08:53:28.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T08:53:28.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:53:28.649 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T08:53:28.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T08:53:28.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:53:28.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:53:28.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:53:28.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:53:28.674 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:53:28.671+0000 7f45c3a0c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:53:28.699 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:53:28.696+0000 7f45c3a0c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:53:28.699 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:53:28.697+0000 7f45c3a0c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:53:29.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T08:53:29.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:53:29.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:53:29.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:53:29.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:53:29.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:53:29.105 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:53:29.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:53:29.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:53:29.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:53:29.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:53:29.727 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:53:29.725+0000 7f45c3a0c500 -1 Falling back to public interface 2023-11-02T08:53:30.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:53:30.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:53:30.477 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:53:30.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:53:30.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:53:30.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:53:30.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:53:31.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:53:31.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:53:31.874 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:53:31.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:53:31.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:53:31.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:53:32.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:53:33.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:53:33.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:53:33.300 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:53:33.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:53:33.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:53:33.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:53:33.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:53:34.165 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:53:34.162+0000 7f45c3a0c500 -1 osd.0 0 log_to_monitors true 2023-11-02T08:53:34.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:53:34.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:53:34.709 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:53:34.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:53:34.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:53:34.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:53:35.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:53:36.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:53:36.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:53:36.176 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:53:36.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:53:36.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:53:36.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:53:36.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:53:37.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:53:37.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:53:37.568 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:53:37.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:53:37.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:53:37.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:53:37.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:53:38.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:53:38.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:53:38.953 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:53:38.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:53:38.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:53:38.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:53:39.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:53:40.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:53:40.351 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:53:40.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:53:40.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:53:40.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:53:40.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:53:40.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:53:41.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:53:41.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:53:41.870 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:53:41.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:53:41.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:53:41.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:53:42.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:53:43.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:53:43.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:53:43.264 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:53:43.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:53:43.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:53:43.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:53:43.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:53:44.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:53:44.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:53:44.832 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:53:44.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:53:44.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:53:44.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:53:45.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:53:46.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:53:46.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:53:46.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:53:46.594 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:53:46.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:53:46.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:53:47.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:53:48.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:53:48.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:53:48.384 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:53:48.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:53:48.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:53:48.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:53:49.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:53:50.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:53:50.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:53:50.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:53:50.950 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:53:50.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:53:50.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:53:51.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:53:52.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:53:52.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:53:52.841 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:53:52.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:53:52.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:53:52.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:53:53.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:53:54.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:53:54.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:53:54.611 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:53:54.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:53:54.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:53:54.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:53:55.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:53:56.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:53:56.286 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T08:53:56.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:53:56.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T08:53:56.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:53:56.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:53:56.725 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/155218,v1:127.0.0.1:6803/155218] [v2:127.0.0.1:6804/155218,v1:127.0.0.1:6805/155218] exists,up b30aad1c-508b-4a9d-871f-a21c604cf0a2 2023-11-02T08:53:56.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:53:56.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:53:56.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:53:56.729 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2023-11-02T08:53:56.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=155218 2023-11-02T08:53:56.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2023-11-02T08:53:56.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2023-11-02T08:53:56.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:53:56.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-11-02T08:53:56.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:53:56.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2023-11-02T08:53:56.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2023-11-02T08:53:56.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:53:56.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:53:56.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:53:56.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2023-11-02T08:53:56.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2023-11-02T08:53:56.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:53:56.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:53:56.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2023-11-02T08:53:56.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:53:56.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:53:56.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:53:56.733 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:53:56.733 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:53:56.733 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T08:53:56.734 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:53:56.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:53:56.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:53:56.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:53:56.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:53:56.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2023-11-02T08:53:56.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2023-11-02T08:53:56.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:53:56.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:53:56.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:53:56.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:53:56.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:53:56.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:53:56.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2023-11-02T08:53:56.737 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:53:56.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d768ec0d-d275-4bc7-a822-c28b6a6e422a 2023-11-02T08:53:56.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d768ec0d-d275-4bc7-a822-c28b6a6e422a' 2023-11-02T08:53:56.738 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 d768ec0d-d275-4bc7-a822-c28b6a6e422a 2023-11-02T08:53:56.738 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:53:56.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCkY0NlKCctLBAApGouyP+vGsnaANZ0OKX3vQ== 2023-11-02T08:53:56.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCkY0NlKCctLBAApGouyP+vGsnaANZ0OKX3vQ=="}' 2023-11-02T08:53:56.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d768ec0d-d275-4bc7-a822-c28b6a6e422a -i td/osd-bluefs-volume-ops/1/new.json 2023-11-02T08:53:57.326 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:53:57.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2023-11-02T08:53:57.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCkY0NlKCctLBAApGouyP+vGsnaANZ0OKX3vQ== --osd-uuid d768ec0d-d275-4bc7-a822-c28b6a6e422a 2023-11-02T08:53:57.362 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:53:57.359+0000 7f7bdf467500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:53:57.363 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:53:57.361+0000 7f7bdf467500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:53:57.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:53:57.361+0000 7f7bdf467500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:53:57.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:53:57.362+0000 7f7bdf467500 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2023-11-02T08:53:57.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:53:57.362+0000 7f7bdf467500 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2023-11-02T08:53:57.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:53:57.362+0000 7f7bdf467500 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2023-11-02T08:53:57.398 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:53:57.395+0000 7f7bdf467500 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2023-11-02T08:54:06.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2023-11-02T08:54:06.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:54:06.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-11-02T08:54:06.641 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2023-11-02T08:54:06.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:54:07.154 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T08:54:07.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-11-02T08:54:07.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:54:07.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:54:07.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:54:07.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:54:07.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:54:07.176+0000 7fbd0fa05500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:54:07.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:54:07.183+0000 7fbd0fa05500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:54:07.187 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:54:07.184+0000 7fbd0fa05500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:54:07.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-11-02T08:54:07.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:54:07.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:54:07.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:54:07.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:54:07.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:07.592 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:54:07.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:54:07.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:07.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:54:08.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:08.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:54:08.213+0000 7fbd0fa05500 -1 Falling back to public interface 2023-11-02T08:54:09.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:09.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:09.002 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:54:09.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:54:09.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:09.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:54:09.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:10.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:10.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:10.412 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:54:10.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:54:10.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:10.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:54:10.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:11.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:11.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:11.829 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:54:11.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:54:11.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:11.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:54:12.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:12.762 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:54:12.759+0000 7fbd0fa05500 -1 osd.1 0 log_to_monitors true 2023-11-02T08:54:13.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:13.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:13.231 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:54:13.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:54:13.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:13.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:54:13.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:14.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:14.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:14.665 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:54:14.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:54:14.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:14.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:54:15.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:16.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:16.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:16.060 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:54:16.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:54:16.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:16.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:54:16.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:17.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:17.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:17.497 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:54:17.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:54:17.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:17.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:54:17.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:18.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:18.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:18.972 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:54:18.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:54:18.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:18.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:54:19.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:20.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:20.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:20.391 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:54:20.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:54:20.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:20.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:54:20.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:21.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:21.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:21.814 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:54:21.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:54:21.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:21.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:54:22.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:23.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:23.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:23.200 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:54:23.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:54:23.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:23.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:54:23.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:24.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:24.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:24.631 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:54:24.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:54:24.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:24.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:54:25.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:26.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:26.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:26.239 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:54:26.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:54:26.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:26.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:54:26.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:27.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:27.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:27.992 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:54:27.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:54:27.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:27.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:54:28.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:29.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:29.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:29.800 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:54:29.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:54:29.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:29.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:54:30.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:31.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:31.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:31.648 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:54:31.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:54:31.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:31.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:54:32.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:33.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:33.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:33.303 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T08:54:33.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T08:54:33.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:33.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:54:34.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:35.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:35.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:35.442 INFO:tasks.workunit.client.0.smithi116.stdout:18 2023-11-02T08:54:35.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2023-11-02T08:54:35.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:35.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:54:35.854 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/157709,v1:127.0.0.1:6811/157709] [v2:127.0.0.1:6812/157709,v1:127.0.0.1:6813/157709] exists,up d768ec0d-d275-4bc7-a822-c28b6a6e422a 2023-11-02T08:54:35.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:54:35.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:54:35.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:54:35.855 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2023-11-02T08:54:35.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=157709 2023-11-02T08:54:35.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2023-11-02T08:54:35.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2023-11-02T08:54:35.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:54:35.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-11-02T08:54:35.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:54:35.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2023-11-02T08:54:35.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2023-11-02T08:54:35.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:54:35.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:54:35.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:54:35.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2023-11-02T08:54:35.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2023-11-02T08:54:35.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:54:35.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:54:35.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2023-11-02T08:54:35.863 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:54:35.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:54:35.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:54:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:54:35.865 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:54:35.865 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T08:54:35.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:54:35.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:54:35.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:54:35.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:54:35.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:54:35.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2023-11-02T08:54:35.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2023-11-02T08:54:35.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:54:35.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:54:35.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:54:35.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:54:35.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:54:35.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:54:35.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2023-11-02T08:54:35.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:54:35.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3b1ee458-46df-41df-a8cd-b9dff9a245c8 2023-11-02T08:54:35.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 3b1ee458-46df-41df-a8cd-b9dff9a245c8' 2023-11-02T08:54:35.869 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 3b1ee458-46df-41df-a8cd-b9dff9a245c8 2023-11-02T08:54:35.870 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:54:35.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDLY0Nl2rX9MxAA8NvF3+tIKkg2k9riBWOElA== 2023-11-02T08:54:35.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDLY0Nl2rX9MxAA8NvF3+tIKkg2k9riBWOElA=="}' 2023-11-02T08:54:35.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3b1ee458-46df-41df-a8cd-b9dff9a245c8 -i td/osd-bluefs-volume-ops/2/new.json 2023-11-02T08:54:36.404 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:54:36.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2023-11-02T08:54:36.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDLY0Nl2rX9MxAA8NvF3+tIKkg2k9riBWOElA== --osd-uuid 3b1ee458-46df-41df-a8cd-b9dff9a245c8 2023-11-02T08:54:36.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:54:36.439+0000 7f682a214500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:54:36.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:54:36.441+0000 7f682a214500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:54:36.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:54:36.441+0000 7f682a214500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:54:36.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:54:36.442+0000 7f682a214500 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2023-11-02T08:54:36.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:54:36.442+0000 7f682a214500 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2023-11-02T08:54:36.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:54:36.442+0000 7f682a214500 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2023-11-02T08:54:36.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:54:36.484+0000 7f682a214500 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2023-11-02T08:54:45.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2023-11-02T08:54:45.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:54:45.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-11-02T08:54:45.761 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2023-11-02T08:54:45.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:54:46.285 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T08:54:46.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-11-02T08:54:46.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:54:46.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:54:46.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:54:46.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:54:46.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:54:46.307+0000 7f8346b39500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:54:46.335 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:54:46.332+0000 7f8346b39500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:54:46.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:54:46.333+0000 7f8346b39500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:54:46.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-11-02T08:54:46.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:54:46.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T08:54:46.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:54:46.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:54:46.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:46.733 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:54:46.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:54:46.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:46.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:54:47.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:47.367 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:54:47.364+0000 7f8346b39500 -1 Falling back to public interface 2023-11-02T08:54:48.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:48.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:48.139 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:54:48.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:54:48.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:48.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:54:48.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:49.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:49.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:49.513 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:54:49.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:54:49.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:49.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:54:49.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:50.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:50.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:50.937 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:54:50.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:54:50.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:50.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:54:51.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:51.885 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:54:51.882+0000 7f8346b39500 -1 osd.2 0 log_to_monitors true 2023-11-02T08:54:52.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:52.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:52.326 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:54:52.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:54:52.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:52.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:54:52.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:53.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:53.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:53.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:54:53.708 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:54:53.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:53.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:54:54.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:55.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:55.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:55.199 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:54:55.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:54:55.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:55.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:54:55.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:56.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:56.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:56.640 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:54:56.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:54:56.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:56.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:54:57.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:58.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:58.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:58.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:54:58.046 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:54:58.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:58.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:54:58.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:54:59.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:54:59.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:54:59.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:54:59.451 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:54:59.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:54:59.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:54:59.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:00.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:00.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:00.891 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:55:00.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:55:00.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:00.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:55:01.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:02.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:02.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:02.299 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:55:02.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:55:02.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:02.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:55:02.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:03.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:03.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:03.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:55:03.770 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:55:03.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:03.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:55:04.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:05.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:05.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:05.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:55:05.287 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:55:05.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:05.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:55:06.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:07.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:07.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:07.060 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:55:07.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:55:07.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:07.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:55:08.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:09.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:09.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:09.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:55:09.433 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:55:09.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:09.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:55:10.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:11.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:11.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:11.462 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:55:11.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:55:11.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:11.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:55:12.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:13.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:13.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:13.158 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T08:55:13.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T08:55:13.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:13.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:55:14.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:15.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:15.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:15.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2023-11-02T08:55:15.063 INFO:tasks.workunit.client.0.smithi116.stdout:18 2023-11-02T08:55:15.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:15.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:55:15.495 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/160225,v1:127.0.0.1:6819/160225] [v2:127.0.0.1:6820/160225,v1:127.0.0.1:6821/160225] exists,up 3b1ee458-46df-41df-a8cd-b9dff9a245c8 2023-11-02T08:55:15.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:55:15.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:55:15.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:55:15.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2023-11-02T08:55:15.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=160225 2023-11-02T08:55:15.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2023-11-02T08:55:15.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2023-11-02T08:55:15.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T08:55:15.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2023-11-02T08:55:15.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T08:55:15.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2023-11-02T08:55:15.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2023-11-02T08:55:15.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:55:15.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:55:15.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:55:15.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2023-11-02T08:55:15.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2023-11-02T08:55:15.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T08:55:15.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T08:55:15.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2023-11-02T08:55:15.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T08:55:15.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:55:15.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:55:15.505 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:55:15.505 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:55:15.506 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T08:55:15.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:55:15.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:55:15.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:55:15.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T08:55:15.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T08:55:15.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2023-11-02T08:55:15.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2023-11-02T08:55:15.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:55:15.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:55:15.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:55:15.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:55:15.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T08:55:15.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T08:55:15.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2023-11-02T08:55:15.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T08:55:15.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=22ca1871-a3a0-4021-b864-063cde3e60c0 2023-11-02T08:55:15.509 INFO:tasks.workunit.client.0.smithi116.stdout:add osd3 22ca1871-a3a0-4021-b864-063cde3e60c0 2023-11-02T08:55:15.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 22ca1871-a3a0-4021-b864-063cde3e60c0' 2023-11-02T08:55:15.510 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T08:55:15.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDzY0NlI5OgHhAAQQWUWqFQhOobcb9Hb7SUWA== 2023-11-02T08:55:15.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDzY0NlI5OgHhAAQQWUWqFQhOobcb9Hb7SUWA=="}' 2023-11-02T08:55:15.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 22ca1871-a3a0-4021-b864-063cde3e60c0 -i td/osd-bluefs-volume-ops/3/new.json 2023-11-02T08:55:16.012 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:55:16.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2023-11-02T08:55:16.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDzY0NlI5OgHhAAQQWUWqFQhOobcb9Hb7SUWA== --osd-uuid 22ca1871-a3a0-4021-b864-063cde3e60c0 2023-11-02T08:55:16.049 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:55:16.046+0000 7f51eb992500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:55:16.051 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:55:16.048+0000 7f51eb992500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:55:16.052 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:55:16.049+0000 7f51eb992500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:55:16.052 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:55:16.049+0000 7f51eb992500 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2023-11-02T08:55:16.052 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:55:16.049+0000 7f51eb992500 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2023-11-02T08:55:16.053 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:55:16.049+0000 7f51eb992500 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2023-11-02T08:55:16.152 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:55:16.150+0000 7f51eb992500 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2023-11-02T08:55:25.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2023-11-02T08:55:25.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T08:55:25.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2023-11-02T08:55:25.105 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd3 key to auth repository 2023-11-02T08:55:25.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T08:55:25.615 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.3 2023-11-02T08:55:25.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2023-11-02T08:55:25.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:55:25.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T08:55:25.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T08:55:25.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T08:55:25.640 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:55:25.636+0000 7f7bc0572500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:55:25.700 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:55:25.696+0000 7f7bc0572500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:55:25.701 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:55:25.698+0000 7f7bc0572500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:55:26.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2023-11-02T08:55:26.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:55:26.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2023-11-02T08:55:26.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:55:26.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:55:26.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:26.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:55:26.040 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:55:26.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:26.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:55:26.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:26.732 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:55:26.727+0000 7f7bc0572500 -1 Falling back to public interface 2023-11-02T08:55:27.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:27.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:27.410 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:55:27.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:55:27.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:27.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:55:27.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:28.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:28.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:28.801 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:55:28.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:55:28.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:28.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:55:29.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:30.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:30.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:30.224 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:55:30.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:55:30.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:30.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:55:30.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:31.353 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:55:31.350+0000 7f7bc0572500 -1 osd.3 0 log_to_monitors true 2023-11-02T08:55:31.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:31.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:31.647 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:55:31.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:55:31.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:31.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:55:32.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:33.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:33.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:33.176 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T08:55:33.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T08:55:33.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:33.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:55:33.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:34.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:34.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:34.634 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T08:55:34.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T08:55:34.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:34.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:55:35.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:36.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:36.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:36.106 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T08:55:36.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T08:55:36.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:36.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:55:36.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:37.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:37.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:37.489 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T08:55:37.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T08:55:37.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:37.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:55:37.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:38.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:38.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:38.904 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T08:55:38.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T08:55:38.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:38.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:55:39.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:40.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:40.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:40.344 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T08:55:40.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T08:55:40.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:40.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:55:40.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:41.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:41.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:41.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T08:55:41.747 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T08:55:41.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:41.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:55:42.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:43.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:43.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:43.261 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T08:55:43.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T08:55:43.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:43.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:55:44.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:45.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:45.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:45.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T08:55:45.050 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T08:55:45.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:45.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:55:45.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:46.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:46.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:46.838 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T08:55:46.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T08:55:46.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:46.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:55:47.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:48.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:48.451 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T08:55:48.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:48.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T08:55:48.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:48.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:55:49.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:50.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:50.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:50.171 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T08:55:50.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T08:55:50.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:50.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:55:51.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:52.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:52.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:52.104 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T08:55:52.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T08:55:52.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:52.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:55:52.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:55:53.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:55:53.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:55:53.965 INFO:tasks.workunit.client.0.smithi116.stdout:18 2023-11-02T08:55:53.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2023-11-02T08:55:53.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:55:53.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:55:54.379 INFO:tasks.workunit.client.0.smithi116.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/162758,v1:127.0.0.1:6827/162758] [v2:127.0.0.1:6828/162758,v1:127.0.0.1:6829/162758] exists,up 22ca1871-a3a0-4021-b864-063cde3e60c0 2023-11-02T08:55:54.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:55:54.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:55:54.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:55:54.381 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2023-11-02T08:55:54.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=162758 2023-11-02T08:55:54.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2023-11-02T08:55:59.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2023-11-02T08:55:59.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2023-11-02T08:55:59.976 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'foo' already exists 2023-11-02T08:55:59.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T08:56:00.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2023-11-02T08:56:01.022 INFO:tasks.workunit.client.0.smithi116.stdout:hints = 1 2023-11-02T08:56:01.023 INFO:tasks.workunit.client.0.smithi116.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2023-11-02T08:56:01.023 INFO:tasks.workunit.client.0.smithi116.stdout:Object prefix: benchmark_data_smithi116_164072 2023-11-02T08:56:01.023 INFO:tasks.workunit.client.0.smithi116.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-11-02T08:56:01.023 INFO:tasks.workunit.client.0.smithi116.stdout: 0 0 0 0 0 0 - 0 2023-11-02T08:56:02.023 INFO:tasks.workunit.client.0.smithi116.stdout: 1 16 39 23 0.0898336 0.0898438 0.301925 0.563495 2023-11-02T08:56:03.023 INFO:tasks.workunit.client.0.smithi116.stdout: 2 16 89 73 0.14256 0.195312 0.304165 0.398008 2023-11-02T08:56:04.023 INFO:tasks.workunit.client.0.smithi116.stdout: 3 16 143 127 0.165344 0.210938 0.29838 0.360856 2023-11-02T08:56:05.023 INFO:tasks.workunit.client.0.smithi116.stdout: 4 16 199 183 0.178688 0.21875 0.306913 0.336205 2023-11-02T08:56:06.024 INFO:tasks.workunit.client.0.smithi116.stdout: 5 16 261 245 0.191381 0.242188 0.284957 0.318244 2023-11-02T08:56:07.023 INFO:tasks.workunit.client.0.smithi116.stdout: 6 16 325 309 0.201144 0.25 0.243793 0.304139 2023-11-02T08:56:08.024 INFO:tasks.workunit.client.0.smithi116.stdout: 7 16 391 375 0.209234 0.257812 0.229127 0.295113 2023-11-02T08:56:09.024 INFO:tasks.workunit.client.0.smithi116.stdout: 8 16 455 439 0.214325 0.25 0.249198 0.289026 2023-11-02T08:56:10.024 INFO:tasks.workunit.client.0.smithi116.stdout: 9 16 512 496 0.215248 0.222656 0.276683 0.284378 2023-11-02T08:56:11.024 INFO:tasks.workunit.client.0.smithi116.stdout: 10 16 574 558 0.217939 0.242188 0.245971 0.28254 2023-11-02T08:56:12.024 INFO:tasks.workunit.client.0.smithi116.stdout: 11 16 630 614 0.218011 0.21875 0.247592 0.283594 2023-11-02T08:56:13.024 INFO:tasks.workunit.client.0.smithi116.stdout: 12 16 689 673 0.219046 0.230469 0.251488 0.281893 2023-11-02T08:56:14.024 INFO:tasks.workunit.client.0.smithi116.stdout: 13 16 750 734 0.220523 0.238281 0.260222 0.280764 2023-11-02T08:56:15.024 INFO:tasks.workunit.client.0.smithi116.stdout: 14 16 812 796 0.222068 0.242188 0.267619 0.278549 2023-11-02T08:56:16.024 INFO:tasks.workunit.client.0.smithi116.stdout: 15 16 870 854 0.222366 0.226562 0.252212 0.278561 2023-11-02T08:56:17.024 INFO:tasks.workunit.client.0.smithi116.stdout: 16 16 933 917 0.223848 0.246094 0.251375 0.276783 2023-11-02T08:56:18.025 INFO:tasks.workunit.client.0.smithi116.stdout: 17 16 991 975 0.224006 0.226562 0.276757 0.276666 2023-11-02T08:56:19.025 INFO:tasks.workunit.client.0.smithi116.stdout: 18 16 1058 1042 0.2261 0.261719 0.211021 0.275202 2023-11-02T08:56:20.025 INFO:tasks.workunit.client.0.smithi116.stdout: 19 16 1104 1088 0.223657 0.179688 0.427378 0.276339 2023-11-02T08:56:21.025 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:56:21.023373+0000 min lat: 0.193169 max lat: 0.815706 avg lat: 0.277836 2023-11-02T08:56:21.026 INFO:tasks.workunit.client.0.smithi116.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-11-02T08:56:21.026 INFO:tasks.workunit.client.0.smithi116.stdout: 20 16 1157 1141 0.222824 0.207031 0.319345 0.277836 2023-11-02T08:56:22.025 INFO:tasks.workunit.client.0.smithi116.stdout: 21 16 1214 1198 0.222814 0.222656 0.327167 0.279156 2023-11-02T08:56:23.025 INFO:tasks.workunit.client.0.smithi116.stdout: 22 16 1274 1258 0.223338 0.234375 0.268618 0.278103 2023-11-02T08:56:24.025 INFO:tasks.workunit.client.0.smithi116.stdout: 23 16 1345 1329 0.225685 0.277344 0.188891 0.275713 2023-11-02T08:56:25.026 INFO:tasks.workunit.client.0.smithi116.stdout: 24 16 1409 1393 0.226697 0.25 0.275384 0.274316 2023-11-02T08:56:26.026 INFO:tasks.workunit.client.0.smithi116.stdout: 25 16 1470 1454 0.227159 0.238281 0.285436 0.273535 2023-11-02T08:56:27.026 INFO:tasks.workunit.client.0.smithi116.stdout: 26 16 1532 1516 0.227736 0.242188 0.310492 0.273112 2023-11-02T08:56:28.026 INFO:tasks.workunit.client.0.smithi116.stdout: 27 16 1589 1573 0.227547 0.222656 0.267104 0.273014 2023-11-02T08:56:29.026 INFO:tasks.workunit.client.0.smithi116.stdout: 28 16 1648 1632 0.22765 0.230469 0.293477 0.272903 2023-11-02T08:56:30.026 INFO:tasks.workunit.client.0.smithi116.stdout: 29 16 1706 1690 0.227612 0.226562 0.259872 0.273405 2023-11-02T08:56:31.026 INFO:tasks.workunit.client.0.smithi116.stdout: 30 13 1765 1752 0.228097 0.242188 0.260354 0.273206 2023-11-02T08:56:32.026 INFO:tasks.workunit.client.0.smithi116.stdout:Total time run: 30.2198 2023-11-02T08:56:32.027 INFO:tasks.workunit.client.0.smithi116.stdout:Total writes made: 1765 2023-11-02T08:56:32.027 INFO:tasks.workunit.client.0.smithi116.stdout:Write size: 4096 2023-11-02T08:56:32.027 INFO:tasks.workunit.client.0.smithi116.stdout:Object size: 4096 2023-11-02T08:56:32.027 INFO:tasks.workunit.client.0.smithi116.stdout:Bandwidth (MB/sec): 0.228146 2023-11-02T08:56:32.028 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev Bandwidth: 0.0326917 2023-11-02T08:56:32.028 INFO:tasks.workunit.client.0.smithi116.stdout:Max bandwidth (MB/sec): 0.277344 2023-11-02T08:56:32.028 INFO:tasks.workunit.client.0.smithi116.stdout:Min bandwidth (MB/sec): 0.0898438 2023-11-02T08:56:32.028 INFO:tasks.workunit.client.0.smithi116.stdout:Average IOPS: 58 2023-11-02T08:56:32.028 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev IOPS: 8.36907 2023-11-02T08:56:32.029 INFO:tasks.workunit.client.0.smithi116.stdout:Max IOPS: 71 2023-11-02T08:56:32.029 INFO:tasks.workunit.client.0.smithi116.stdout:Min IOPS: 23 2023-11-02T08:56:32.029 INFO:tasks.workunit.client.0.smithi116.stdout:Average Latency(s): 0.273063 2023-11-02T08:56:32.029 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev Latency(s): 0.0562941 2023-11-02T08:56:32.029 INFO:tasks.workunit.client.0.smithi116.stdout:Max latency(s): 0.815706 2023-11-02T08:56:32.029 INFO:tasks.workunit.client.0.smithi116.stdout:Min latency(s): 0.180153 2023-11-02T08:56:32.330 INFO:tasks.workunit.client.0.smithi116.stdout:after bench 2023-11-02T08:56:32.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2023-11-02T08:56:32.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 155218 2023-11-02T08:56:32.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2023-11-02T08:56:33.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 155218 2023-11-02T08:56:33.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (155218) - No such process 2023-11-02T08:56:33.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2023-11-02T08:56:33.752 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 is already down. 2023-11-02T08:56:33.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 157709 2023-11-02T08:56:33.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2023-11-02T08:56:34.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 157709 2023-11-02T08:56:34.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (157709) - No such process 2023-11-02T08:56:34.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2023-11-02T08:56:35.209 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 is already down. 2023-11-02T08:56:35.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 160225 2023-11-02T08:56:35.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2023-11-02T08:56:36.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 160225 2023-11-02T08:56:36.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (160225) - No such process 2023-11-02T08:56:36.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2023-11-02T08:56:36.633 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 is already down. 2023-11-02T08:56:36.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 162758 2023-11-02T08:56:36.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2023-11-02T08:56:37.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 162758 2023-11-02T08:56:37.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (162758) - No such process 2023-11-02T08:56:37.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2023-11-02T08:56:38.164 INFO:tasks.workunit.client.0.smithi116.stderr:osd.3 is already down. 2023-11-02T08:56:38.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2023-11-02T08:56:39.956 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:56:39.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2023-11-02T08:56:41.752 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:56:41.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2023-11-02T08:56:43.549 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:56:43.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2023-11-02T08:56:45.348 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:56:45.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2023-11-02T08:56:45.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2023-11-02T08:56:45.402 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T08:56:46.221 INFO:tasks.workunit.client.0.smithi116.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2023-11-02T08:56:46.221 INFO:tasks.workunit.client.0.smithi116.stdout:1 : device size 0x3fffe000 : using 0xc00000(12 MiB) 2023-11-02T08:56:46.221 INFO:tasks.workunit.client.0.smithi116.stdout:2 : device size 0x100000000 : using 0x8056f000(2.0 GiB) 2023-11-02T08:56:46.221 INFO:tasks.workunit.client.0.smithi116.stdout:Expanding DB/WAL... 2023-11-02T08:56:46.222 INFO:tasks.workunit.client.0.smithi116.stdout:0 : expanding to 0x536870912 2023-11-02T08:56:46.246 INFO:tasks.workunit.client.0.smithi116.stdout:0 : size label updated to 536870912 2023-11-02T08:56:46.246 INFO:tasks.workunit.client.0.smithi116.stdout:1 : expanding to 0x1073741824 2023-11-02T08:56:46.271 INFO:tasks.workunit.client.0.smithi116.stdout:1 : size label updated to 1073741824 2023-11-02T08:56:46.271 INFO:tasks.workunit.client.0.smithi116.stdout:2 : expanding from 0x80000000 to 0x100000000 2023-11-02T08:56:46.472 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:56:46.469+0000 7f3e341f0800 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/0: (21) Is a directory 2023-11-02T08:56:46.474 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:56:46.469+0000 7f3e341f0800 -1 bluestore(td/osd-bluefs-volume-ops/0) unable to read label for td/osd-bluefs-volume-ops/0: (21) Is a directory 2023-11-02T08:56:51.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2023-11-02T08:56:51.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2023-11-02T08:56:51.417 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T08:56:52.251 INFO:tasks.workunit.client.0.smithi116.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2023-11-02T08:56:52.251 INFO:tasks.workunit.client.0.smithi116.stdout:1 : device size 0x3fffe000 : using 0xc00000(12 MiB) 2023-11-02T08:56:52.252 INFO:tasks.workunit.client.0.smithi116.stdout:2 : device size 0x101000000 : using 0x8155f000(2.0 GiB) 2023-11-02T08:56:52.252 INFO:tasks.workunit.client.0.smithi116.stdout:Expanding DB/WAL... 2023-11-02T08:56:52.252 INFO:tasks.workunit.client.0.smithi116.stdout:0 : expanding to 0x536870912 2023-11-02T08:56:52.269 INFO:tasks.workunit.client.0.smithi116.stdout:0 : size label updated to 536870912 2023-11-02T08:56:52.269 INFO:tasks.workunit.client.0.smithi116.stdout:1 : expanding to 0x1073741824 2023-11-02T08:56:52.294 INFO:tasks.workunit.client.0.smithi116.stdout:1 : size label updated to 1073741824 2023-11-02T08:56:52.294 INFO:tasks.workunit.client.0.smithi116.stdout:2 : expanding from 0x80000000 to 0x101000000 2023-11-02T08:56:52.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:56:52.484+0000 7f8331c10800 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/1: (21) Is a directory 2023-11-02T08:56:52.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:56:52.484+0000 7f8331c10800 -1 bluestore(td/osd-bluefs-volume-ops/1) unable to read label for td/osd-bluefs-volume-ops/1: (21) Is a directory 2023-11-02T08:56:57.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2023-11-02T08:56:57.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2023-11-02T08:56:57.514 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T08:56:58.344 INFO:tasks.workunit.client.0.smithi116.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2023-11-02T08:56:58.344 INFO:tasks.workunit.client.0.smithi116.stdout:1 : device size 0x3fffe000 : using 0xc00000(12 MiB) 2023-11-02T08:56:58.344 INFO:tasks.workunit.client.0.smithi116.stdout:2 : device size 0x100020000 : using 0x80580000(2.0 GiB) 2023-11-02T08:56:58.344 INFO:tasks.workunit.client.0.smithi116.stdout:Expanding DB/WAL... 2023-11-02T08:56:58.344 INFO:tasks.workunit.client.0.smithi116.stdout:0 : expanding to 0x536870912 2023-11-02T08:56:58.367 INFO:tasks.workunit.client.0.smithi116.stdout:0 : size label updated to 536870912 2023-11-02T08:56:58.368 INFO:tasks.workunit.client.0.smithi116.stdout:1 : expanding to 0x1073741824 2023-11-02T08:56:58.392 INFO:tasks.workunit.client.0.smithi116.stdout:1 : size label updated to 1073741824 2023-11-02T08:56:58.393 INFO:tasks.workunit.client.0.smithi116.stdout:2 : expanding from 0x80000000 to 0x100020000 2023-11-02T08:56:58.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:56:58.582+0000 7f3775639800 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/2: (21) Is a directory 2023-11-02T08:56:58.585 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:56:58.582+0000 7f3775639800 -1 bluestore(td/osd-bluefs-volume-ops/2) unable to read label for td/osd-bluefs-volume-ops/2: (21) Is a directory 2023-11-02T08:57:03.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2023-11-02T08:57:03.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2023-11-02T08:57:03.838 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T08:57:04.673 INFO:tasks.workunit.client.0.smithi116.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2023-11-02T08:57:04.674 INFO:tasks.workunit.client.0.smithi116.stdout:1 : device size 0x3fffe000 : using 0xc00000(12 MiB) 2023-11-02T08:57:04.674 INFO:tasks.workunit.client.0.smithi116.stdout:2 : device size 0xfff00000 : using 0x80488000(2.0 GiB) 2023-11-02T08:57:04.674 INFO:tasks.workunit.client.0.smithi116.stdout:Expanding DB/WAL... 2023-11-02T08:57:04.674 INFO:tasks.workunit.client.0.smithi116.stdout:0 : expanding to 0x536870912 2023-11-02T08:57:04.690 INFO:tasks.workunit.client.0.smithi116.stdout:0 : size label updated to 536870912 2023-11-02T08:57:04.691 INFO:tasks.workunit.client.0.smithi116.stdout:1 : expanding to 0x1073741824 2023-11-02T08:57:04.722 INFO:tasks.workunit.client.0.smithi116.stdout:1 : size label updated to 1073741824 2023-11-02T08:57:04.722 INFO:tasks.workunit.client.0.smithi116.stdout:2 : expanding from 0x80000000 to 0xfff00000 2023-11-02T08:57:04.903 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:04.900+0000 7fced423d800 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/3: (21) Is a directory 2023-11-02T08:57:04.904 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:04.900+0000 7fced423d800 -1 bluestore(td/osd-bluefs-volume-ops/3) unable to read label for td/osd-bluefs-volume-ops/3: (21) Is a directory 2023-11-02T08:57:10.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2023-11-02T08:57:11.825 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:57:11.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2023-11-02T08:57:13.619 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:57:13.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2023-11-02T08:57:15.418 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:57:15.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2023-11-02T08:57:17.220 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:57:17.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2023-11-02T08:57:17.250 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T08:57:19.017 INFO:tasks.workunit.client.0.smithi116.stdout:0 : device size 0x1ffff000 : using 0x1600000(22 MiB) 2023-11-02T08:57:19.017 INFO:tasks.workunit.client.0.smithi116.stdout:1 : device size 0x3fffe000 : using 0x1400000(20 MiB) 2023-11-02T08:57:19.017 INFO:tasks.workunit.client.0.smithi116.stdout:2 : device size 0x100000000 : using 0x56f000(5.4 MiB) 2023-11-02T08:57:19.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2023-11-02T08:57:19.040 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T08:57:20.556 INFO:tasks.workunit.client.0.smithi116.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2023-11-02T08:57:20.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2023-11-02T08:57:21.851 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:57:21.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2023-11-02T08:57:21.884 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T08:57:23.647 INFO:tasks.workunit.client.0.smithi116.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2023-11-02T08:57:23.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2023-11-02T08:57:24.942 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:57:24.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2023-11-02T08:57:24.975 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T08:57:26.742 INFO:tasks.workunit.client.0.smithi116.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2023-11-02T08:57:26.742 INFO:tasks.workunit.client.0.smithi116.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2023-11-02T08:57:26.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2023-11-02T08:57:27.533 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:57:27.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2023-11-02T08:57:27.565 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T08:57:27.566 INFO:tasks.workunit.client.0.smithi116.stderr:Migrate to WAL device isn't supported. 2023-11-02T08:57:27.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2023-11-02T08:57:27.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2023-11-02T08:57:29.353 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:57:29.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2023-11-02T08:57:29.386 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T08:57:30.898 INFO:tasks.workunit.client.0.smithi116.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2023-11-02T08:57:30.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2023-11-02T08:57:32.188 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:57:32.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2023-11-02T08:57:32.221 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T08:57:33.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2023-11-02T08:57:34.782 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:57:34.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2023-11-02T08:57:34.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2023-11-02T08:57:34.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:57:34.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-11-02T08:57:34.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:57:34.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2023-11-02T08:57:34.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2023-11-02T08:57:34.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:57:34.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:57:34.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:57:34.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2023-11-02T08:57:34.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2023-11-02T08:57:34.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:57:34.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:57:34.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2023-11-02T08:57:34.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:57:34.800 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:57:34.800 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:57:34.800 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:57:34.800 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:57:34.800 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T08:57:34.800 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:57:34.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:57:34.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:57:34.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2023-11-02T08:57:34.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2023-11-02T08:57:34.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:57:34.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:57:34.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:57:34.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:57:34.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:57:34.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:57:34.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2023-11-02T08:57:34.803 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T08:57:34.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-11-02T08:57:34.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:57:34.805 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2023-11-02T08:57:34.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-11-02T08:57:34.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:57:34.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:57:34.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:57:34.824 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:34.820+0000 7fbc3d0b2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:57:34.826 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:34.822+0000 7fbc3d0b2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:57:34.827 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:34.823+0000 7fbc3d0b2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:57:35.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-11-02T08:57:35.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:57:35.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:57:35.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:57:35.178 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:57:35.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:57:35.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:57:35.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:57:35.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:57:35.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:57:35.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:57:35.599 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:35.595+0000 7fbc3d0b2500 -1 Falling back to public interface 2023-11-02T08:57:36.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:57:36.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:57:36.547 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:57:36.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:57:36.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:57:36.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:57:36.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:57:37.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:57:37.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:57:37.919 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:57:37.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:57:37.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:57:37.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:57:38.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:57:39.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:57:39.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:57:39.286 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:57:39.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:57:39.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:57:39.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:57:39.360 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:39.357+0000 7fbc3d0b2500 -1 osd.0 27 log_to_monitors true 2023-11-02T08:57:39.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:57:40.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:57:40.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:57:40.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:57:40.664 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:57:40.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:57:40.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:57:41.088 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 37 up_thru 37 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/176148,v1:127.0.0.1:6803/176148] [v2:127.0.0.1:6804/176148,v1:127.0.0.1:6805/176148] exists,up b30aad1c-508b-4a9d-871f-a21c604cf0a2 2023-11-02T08:57:41.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:57:41.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:57:41.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:57:41.093 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2023-11-02T08:57:41.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=176148 2023-11-02T08:57:41.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2023-11-02T08:57:41.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2023-11-02T08:57:41.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:57:41.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2023-11-02T08:57:41.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:57:41.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2023-11-02T08:57:41.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2023-11-02T08:57:41.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:57:41.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:57:41.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:57:41.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2023-11-02T08:57:41.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2023-11-02T08:57:41.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:57:41.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:57:41.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2023-11-02T08:57:41.098 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:57:41.098 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:57:41.098 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:57:41.098 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:57:41.099 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:57:41.099 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T08:57:41.099 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:57:41.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:57:41.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:57:41.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2023-11-02T08:57:41.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2023-11-02T08:57:41.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:57:41.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:57:41.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:57:41.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:57:41.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:57:41.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:57:41.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2023-11-02T08:57:41.102 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T08:57:41.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2023-11-02T08:57:41.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:57:41.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2023-11-02T08:57:41.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2023-11-02T08:57:41.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:57:41.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:57:41.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:57:41.117 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:41.114+0000 7fb819337500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:57:41.118 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:41.116+0000 7fb819337500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:57:41.119 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:41.117+0000 7fb819337500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:57:41.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2023-11-02T08:57:41.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:57:41.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:57:41.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:57:41.508 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:57:41.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:57:41.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:57:41.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:57:41.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:57:41.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:57:41.887 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:41.885+0000 7fb819337500 -1 Falling back to public interface 2023-11-02T08:57:41.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:57:42.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:57:42.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:57:42.909 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:57:42.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:57:42.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:57:42.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:57:43.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:57:44.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:57:44.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:57:44.290 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:57:44.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:57:44.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:57:44.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:57:44.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:57:45.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:45.440+0000 7fb819337500 -1 osd.1 29 log_to_monitors true 2023-11-02T08:57:45.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:57:45.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:57:45.694 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:57:45.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:57:45.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:57:45.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:57:46.155 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 30 last_clean_interval [10,29) [v2:127.0.0.1:6810/177112,v1:127.0.0.1:6811/177112] [v2:127.0.0.1:6812/177112,v1:127.0.0.1:6813/177112] exists,up d768ec0d-d275-4bc7-a822-c28b6a6e422a 2023-11-02T08:57:46.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:57:46.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:57:46.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:57:46.158 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2023-11-02T08:57:46.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=177112 2023-11-02T08:57:46.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2023-11-02T08:57:46.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2023-11-02T08:57:46.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:57:46.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2023-11-02T08:57:46.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:57:46.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2023-11-02T08:57:46.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2023-11-02T08:57:46.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:57:46.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:57:46.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:57:46.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2023-11-02T08:57:46.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2023-11-02T08:57:46.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:57:46.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:57:46.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2023-11-02T08:57:46.172 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:57:46.173 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:57:46.173 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:57:46.173 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:57:46.173 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:57:46.174 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T08:57:46.174 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:57:46.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:57:46.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:57:46.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2023-11-02T08:57:46.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2023-11-02T08:57:46.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:57:46.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:57:46.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:57:46.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:57:46.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:57:46.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:57:46.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2023-11-02T08:57:46.177 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T08:57:46.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2023-11-02T08:57:46.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:57:46.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2023-11-02T08:57:46.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2023-11-02T08:57:46.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:57:46.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:57:46.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:57:46.192 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:46.189+0000 7efd1e605500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:57:46.194 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:46.191+0000 7efd1e605500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:57:46.195 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:46.192+0000 7efd1e605500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:57:46.714 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:46.711+0000 7efd1e605500 -1 Falling back to public interface 2023-11-02T08:57:46.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2023-11-02T08:57:46.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:57:46.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T08:57:46.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:57:46.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:57:46.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:57:46.785 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:57:46.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:57:46.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:57:46.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:57:47.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:57:48.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:57:48.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:57:48.173 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:57:48.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:57:48.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:57:48.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:57:48.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:57:49.522 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:49.519+0000 7efd1e605500 -1 osd.2 32 log_to_monitors true 2023-11-02T08:57:49.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:57:49.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:57:49.557 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:57:49.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:57:49.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:57:49.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:57:50.181 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 44 up_thru 30 down_at 33 last_clean_interval [15,32) [v2:127.0.0.1:6818/178051,v1:127.0.0.1:6819/178051] [v2:127.0.0.1:6820/178051,v1:127.0.0.1:6821/178051] exists,up 3b1ee458-46df-41df-a8cd-b9dff9a245c8 2023-11-02T08:57:50.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:57:50.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:57:50.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:57:50.186 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2023-11-02T08:57:50.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=178051 2023-11-02T08:57:50.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2023-11-02T08:57:50.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2023-11-02T08:57:50.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:57:50.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2023-11-02T08:57:50.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:57:50.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2023-11-02T08:57:50.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2023-11-02T08:57:50.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:57:50.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:57:50.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:57:50.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2023-11-02T08:57:50.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2023-11-02T08:57:50.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:57:50.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:57:50.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2023-11-02T08:57:50.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:57:50.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:57:50.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:57:50.191 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:57:50.192 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:57:50.192 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T08:57:50.192 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:57:50.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:57:50.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:57:50.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2023-11-02T08:57:50.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2023-11-02T08:57:50.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:57:50.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:57:50.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:57:50.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:57:50.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:57:50.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:57:50.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2023-11-02T08:57:50.196 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.3 2023-11-02T08:57:50.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2023-11-02T08:57:50.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:57:50.197 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2023-11-02T08:57:50.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2023-11-02T08:57:50.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:57:50.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:57:50.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:57:50.209 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:50.206+0000 7fea80970500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:57:50.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:50.208+0000 7fea80970500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:57:50.212 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:50.209+0000 7fea80970500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:57:50.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2023-11-02T08:57:50.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:57:50.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2023-11-02T08:57:50.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:57:50.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:57:50.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:57:50.829 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:57:50.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:57:50.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:57:50.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:57:50.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:50.977+0000 7fea80970500 -1 Falling back to public interface 2023-11-02T08:57:51.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:57:52.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:57:52.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:57:52.423 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:57:52.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:57:52.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:57:52.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:57:52.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:57:53.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:57:53.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:57:53.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:57:53.849 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:57:53.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:57:53.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:57:54.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:57:54.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:57:54.737+0000 7fea80970500 -1 osd.3 34 log_to_monitors true 2023-11-02T08:57:55.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:57:55.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:57:55.242 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:57:55.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:57:55.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:57:55.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:57:55.956 INFO:tasks.workunit.client.0.smithi116.stdout:osd.3 up in weight 1 up_from 48 up_thru 33 down_at 35 last_clean_interval [20,34) [v2:127.0.0.1:6826/178960,v1:127.0.0.1:6827/178960] [v2:127.0.0.1:6828/178960,v1:127.0.0.1:6829/178960] exists,up 22ca1871-a3a0-4021-b864-063cde3e60c0 2023-11-02T08:57:55.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:57:55.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:57:55.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:57:55.958 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2023-11-02T08:57:55.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=178960 2023-11-02T08:57:55.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2023-11-02T08:57:55.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T08:57:55.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T08:57:55.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T08:57:55.968 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-11-02T08:57:55.968 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T08:57:55.968 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T08:57:55.968 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T08:57:55.968 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T08:57:55.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T08:57:56.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-11-02T08:57:56.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T08:57:56.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T08:57:56.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T08:57:56.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T08:57:56.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T08:57:56.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T08:57:56.609 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T08:57:56.609 INFO:tasks.workunit.client.0.smithi116.stderr:2 2023-11-02T08:57:56.610 INFO:tasks.workunit.client.0.smithi116.stderr:3' 2023-11-02T08:57:56.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T08:57:56.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:57:56.610 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T08:57:56.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=158913789958 2023-11-02T08:57:56.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 158913789958 2023-11-02T08:57:56.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-158913789958' 2023-11-02T08:57:56.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:57:56.729 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T08:57:56.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691845 2023-11-02T08:57:56.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691845 2023-11-02T08:57:56.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-158913789958 1-171798691845' 2023-11-02T08:57:56.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:57:56.880 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T08:57:56.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=188978561028 2023-11-02T08:57:56.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 188978561028 2023-11-02T08:57:56.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-158913789958 1-171798691845 2-188978561028' 2023-11-02T08:57:56.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T08:57:56.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-11-02T08:57:57.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=206158430211 2023-11-02T08:57:57.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 206158430211 2023-11-02T08:57:57.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-158913789958 1-171798691845 2-188978561028 3-206158430211' 2023-11-02T08:57:57.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:57:57.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-158913789958 2023-11-02T08:57:57.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:57:57.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T08:57:57.114 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-158913789958 2023-11-02T08:57:57.114 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:57:57.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=158913789958 2023-11-02T08:57:57.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 158913789958' 2023-11-02T08:57:57.115 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 158913789958 2023-11-02T08:57:57.116 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T08:57:57.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 158913789958 -lt 158913789958 2023-11-02T08:57:57.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:57:57.483 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-171798691845 2023-11-02T08:57:57.483 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:57:57.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T08:57:57.484 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-171798691845 2023-11-02T08:57:57.484 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:57:57.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691845 2023-11-02T08:57:57.485 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 171798691845 2023-11-02T08:57:57.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2023-11-02T08:57:57.485 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T08:57:57.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691843 -lt 171798691845 2023-11-02T08:57:57.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T08:57:58.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T08:57:58.849 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T08:57:59.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691845 -lt 171798691845 2023-11-02T08:57:59.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:57:59.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-188978561028 2023-11-02T08:57:59.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:57:59.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T08:57:59.244 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-188978561028 2023-11-02T08:57:59.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:57:59.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=188978561028 2023-11-02T08:57:59.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 188978561028' 2023-11-02T08:57:59.246 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 188978561028 2023-11-02T08:57:59.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T08:57:59.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 188978561028 -lt 188978561028 2023-11-02T08:57:59.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T08:57:59.676 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-206158430211 2023-11-02T08:57:59.676 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T08:57:59.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2023-11-02T08:57:59.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-206158430211 2023-11-02T08:57:59.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T08:57:59.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=206158430211 2023-11-02T08:57:59.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 206158430211' 2023-11-02T08:57:59.679 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 206158430211 2023-11-02T08:57:59.679 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2023-11-02T08:58:00.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 206158430211 -lt 206158430211 2023-11-02T08:58:00.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T08:58:00.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:58:00.086 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:58:00.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2023-11-02T08:58:00.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T08:58:00.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T08:58:00.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T08:58:00.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T08:58:00.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T08:58:00.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T08:58:00.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T08:58:00.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2023-11-02T08:58:00.983 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T08:58:00.983 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T08:58:00.983 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T08:58:01.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2023-11-02T08:58:01.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T08:58:01.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T08:58:01.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2023-11-02T08:58:01.549 INFO:tasks.workunit.client.0.smithi116.stdout:hints = 1 2023-11-02T08:58:01.550 INFO:tasks.workunit.client.0.smithi116.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2023-11-02T08:58:01.550 INFO:tasks.workunit.client.0.smithi116.stdout:Object prefix: benchmark_data_smithi116_180267 2023-11-02T08:58:01.550 INFO:tasks.workunit.client.0.smithi116.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-11-02T08:58:01.550 INFO:tasks.workunit.client.0.smithi116.stdout: 0 0 0 0 0 0 - 0 2023-11-02T08:58:02.550 INFO:tasks.workunit.client.0.smithi116.stdout: 1 16 95 79 0.308555 0.308594 0.180217 0.186791 2023-11-02T08:58:03.550 INFO:tasks.workunit.client.0.smithi116.stdout: 2 16 175 159 0.310505 0.3125 0.217813 0.189868 2023-11-02T08:58:04.550 INFO:tasks.workunit.client.0.smithi116.stdout: 3 16 259 243 0.316363 0.328125 0.200578 0.188753 2023-11-02T08:58:05.550 INFO:tasks.workunit.client.0.smithi116.stdout: 4 16 336 320 0.312457 0.300781 0.24304 0.194712 2023-11-02T08:58:06.550 INFO:tasks.workunit.client.0.smithi116.stdout: 5 16 421 405 0.316366 0.332031 0.178937 0.193201 2023-11-02T08:58:07.550 INFO:tasks.workunit.client.0.smithi116.stdout: 6 16 494 478 0.31116 0.285156 0.175077 0.197402 2023-11-02T08:58:08.550 INFO:tasks.workunit.client.0.smithi116.stdout: 7 16 569 553 0.308557 0.292969 0.22111 0.199134 2023-11-02T08:58:09.550 INFO:tasks.workunit.client.0.smithi116.stdout: 8 16 647 631 0.308067 0.304688 0.312833 0.200033 2023-11-02T08:58:10.550 INFO:tasks.workunit.client.0.smithi116.stdout: 9 16 710 694 0.301177 0.246094 0.209064 0.204618 2023-11-02T08:58:11.551 INFO:tasks.workunit.client.0.smithi116.stdout: 10 16 784 768 0.299962 0.289062 0.179912 0.205665 2023-11-02T08:58:12.551 INFO:tasks.workunit.client.0.smithi116.stdout: 11 16 867 851 0.302165 0.324219 0.125993 0.205115 2023-11-02T08:58:13.551 INFO:tasks.workunit.client.0.smithi116.stdout: 12 16 952 936 0.30465 0.332031 0.188302 0.204092 2023-11-02T08:58:14.551 INFO:tasks.workunit.client.0.smithi116.stdout: 13 16 1031 1015 0.304951 0.308594 0.167373 0.203541 2023-11-02T08:58:15.551 INFO:tasks.workunit.client.0.smithi116.stdout: 14 16 1101 1085 0.302698 0.273438 0.456867 0.203861 2023-11-02T08:58:16.551 INFO:tasks.workunit.client.0.smithi116.stdout: 15 16 1181 1165 0.303349 0.3125 0.210727 0.204806 2023-11-02T08:58:17.551 INFO:tasks.workunit.client.0.smithi116.stdout: 16 16 1263 1247 0.304406 0.320312 0.214639 0.203841 2023-11-02T08:58:18.552 INFO:tasks.workunit.client.0.smithi116.stdout: 17 16 1343 1327 0.30488 0.3125 0.217498 0.203406 2023-11-02T08:58:19.552 INFO:tasks.workunit.client.0.smithi116.stdout: 18 16 1426 1410 0.305952 0.324219 0.166611 0.203598 2023-11-02T08:58:20.552 INFO:tasks.workunit.client.0.smithi116.stdout: 19 16 1508 1492 0.306705 0.320312 0.184242 0.203079 2023-11-02T08:58:21.552 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:58:21.550296+0000 min lat: 0.108276 max lat: 0.476879 avg lat: 0.205632 2023-11-02T08:58:21.552 INFO:tasks.workunit.client.0.smithi116.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-11-02T08:58:21.553 INFO:tasks.workunit.client.0.smithi116.stdout: 20 16 1560 1544 0.301525 0.203125 0.24479 0.205632 2023-11-02T08:58:22.552 INFO:tasks.workunit.client.0.smithi116.stdout: 21 16 1650 1634 0.303905 0.351562 0.12435 0.204987 2023-11-02T08:58:23.552 INFO:tasks.workunit.client.0.smithi116.stdout: 22 16 1721 1705 0.302696 0.277344 0.142789 0.205542 2023-11-02T08:58:24.552 INFO:tasks.workunit.client.0.smithi116.stdout: 23 16 1800 1784 0.30295 0.308594 0.167683 0.205419 2023-11-02T08:58:25.552 INFO:tasks.workunit.client.0.smithi116.stdout: 24 16 1876 1860 0.302696 0.296875 0.22553 0.205924 2023-11-02T08:58:26.553 INFO:tasks.workunit.client.0.smithi116.stdout: 25 16 1956 1940 0.303087 0.3125 0.180819 0.20592 2023-11-02T08:58:27.553 INFO:tasks.workunit.client.0.smithi116.stdout: 26 16 2023 2007 0.301495 0.261719 0.174137 0.206408 2023-11-02T08:58:28.553 INFO:tasks.workunit.client.0.smithi116.stdout: 27 16 2104 2088 0.302046 0.316406 0.219904 0.206437 2023-11-02T08:58:29.553 INFO:tasks.workunit.client.0.smithi116.stdout: 28 16 2181 2165 0.302 0.300781 0.232906 0.206484 2023-11-02T08:58:30.553 INFO:tasks.workunit.client.0.smithi116.stdout: 29 16 2255 2239 0.301553 0.289062 0.134548 0.206744 2023-11-02T08:58:31.553 INFO:tasks.workunit.client.0.smithi116.stdout: 30 16 2327 2311 0.300875 0.28125 0.182929 0.206648 2023-11-02T08:58:32.553 INFO:tasks.workunit.client.0.smithi116.stdout:Total time run: 30.1418 2023-11-02T08:58:32.554 INFO:tasks.workunit.client.0.smithi116.stdout:Total writes made: 2327 2023-11-02T08:58:32.554 INFO:tasks.workunit.client.0.smithi116.stdout:Write size: 4096 2023-11-02T08:58:32.554 INFO:tasks.workunit.client.0.smithi116.stdout:Object size: 4096 2023-11-02T08:58:32.554 INFO:tasks.workunit.client.0.smithi116.stdout:Bandwidth (MB/sec): 0.30157 2023-11-02T08:58:32.555 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev Bandwidth: 0.0290418 2023-11-02T08:58:32.555 INFO:tasks.workunit.client.0.smithi116.stdout:Max bandwidth (MB/sec): 0.351562 2023-11-02T08:58:32.555 INFO:tasks.workunit.client.0.smithi116.stdout:Min bandwidth (MB/sec): 0.203125 2023-11-02T08:58:32.555 INFO:tasks.workunit.client.0.smithi116.stdout:Average IOPS: 77 2023-11-02T08:58:32.555 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev IOPS: 7.4347 2023-11-02T08:58:32.556 INFO:tasks.workunit.client.0.smithi116.stdout:Max IOPS: 90 2023-11-02T08:58:32.556 INFO:tasks.workunit.client.0.smithi116.stdout:Min IOPS: 52 2023-11-02T08:58:32.556 INFO:tasks.workunit.client.0.smithi116.stdout:Average Latency(s): 0.206635 2023-11-02T08:58:32.556 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev Latency(s): 0.0445985 2023-11-02T08:58:32.556 INFO:tasks.workunit.client.0.smithi116.stdout:Max latency(s): 0.476879 2023-11-02T08:58:32.556 INFO:tasks.workunit.client.0.smithi116.stdout:Min latency(s): 0.108276 2023-11-02T08:58:32.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 176148 2023-11-02T08:58:32.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2023-11-02T08:58:33.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 176148 2023-11-02T08:58:33.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (176148) - No such process 2023-11-02T08:58:33.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2023-11-02T08:58:34.090 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 is already down. 2023-11-02T08:58:34.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 177112 2023-11-02T08:58:34.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2023-11-02T08:58:35.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 177112 2023-11-02T08:58:35.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (177112) - No such process 2023-11-02T08:58:35.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2023-11-02T08:58:35.539 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 is already down. 2023-11-02T08:58:35.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 178051 2023-11-02T08:58:35.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2023-11-02T08:58:36.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 178051 2023-11-02T08:58:36.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (178051) - No such process 2023-11-02T08:58:36.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2023-11-02T08:58:37.008 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 is already down. 2023-11-02T08:58:37.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 178960 2023-11-02T08:58:37.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2023-11-02T08:58:38.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 178960 2023-11-02T08:58:38.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (178960) - No such process 2023-11-02T08:58:38.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2023-11-02T08:58:38.532 INFO:tasks.workunit.client.0.smithi116.stderr:osd.3 is already down. 2023-11-02T08:58:38.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2023-11-02T08:58:39.824 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:58:39.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2023-11-02T08:58:40.080 INFO:tasks.workunit.client.0.smithi116.stderr:512+0 records in 2023-11-02T08:58:40.080 INFO:tasks.workunit.client.0.smithi116.stderr:512+0 records out 2023-11-02T08:58:40.080 INFO:tasks.workunit.client.0.smithi116.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.246108 s, 2.2 GB/s 2023-11-02T08:58:40.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2023-11-02T08:58:40.099 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T08:58:44.613 INFO:tasks.workunit.client.0.smithi116.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2023-11-02T08:58:44.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2023-11-02T08:58:46.403 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:58:46.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2023-11-02T08:58:47.693 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:58:47.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2023-11-02T08:58:48.201 INFO:tasks.workunit.client.0.smithi116.stderr:1024+0 records in 2023-11-02T08:58:48.201 INFO:tasks.workunit.client.0.smithi116.stderr:1024+0 records out 2023-11-02T08:58:48.202 INFO:tasks.workunit.client.0.smithi116.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.499604 s, 2.1 GB/s 2023-11-02T08:58:48.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2023-11-02T08:58:48.220 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T08:58:55.735 INFO:tasks.workunit.client.0.smithi116.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2023-11-02T08:58:55.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2023-11-02T08:58:55.758 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T08:58:57.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2023-11-02T08:58:59.313 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:58:59.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2023-11-02T08:59:00.105 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:59:00.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2023-11-02T08:59:00.131 INFO:tasks.workunit.client.0.smithi116.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2023-11-02T08:59:00.132 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T08:59:00.894 INFO:tasks.workunit.client.0.smithi116.stdout:DB device added 2023-11-02T08:59:00.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2023-11-02T08:59:00.920 INFO:tasks.workunit.client.0.smithi116.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2023-11-02T08:59:00.920 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T08:59:02.191 INFO:tasks.workunit.client.0.smithi116.stdout:WAL device added 2023-11-02T08:59:02.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2023-11-02T08:59:02.215 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T08:59:04.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2023-11-02T08:59:06.021 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:59:06.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2023-11-02T08:59:07.312 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:59:07.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2023-11-02T08:59:07.341 INFO:tasks.workunit.client.0.smithi116.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2023-11-02T08:59:07.342 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T08:59:08.603 INFO:tasks.workunit.client.0.smithi116.stdout:WAL device added 2023-11-02T08:59:08.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2023-11-02T08:59:08.628 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T08:59:10.641 INFO:tasks.workunit.client.0.smithi116.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2023-11-02T08:59:10.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2023-11-02T08:59:11.930 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T08:59:11.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2023-11-02T08:59:11.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2023-11-02T08:59:11.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:59:11.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-11-02T08:59:11.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:59:11.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2023-11-02T08:59:11.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2023-11-02T08:59:11.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:59:11.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:59:11.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:59:11.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2023-11-02T08:59:11.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2023-11-02T08:59:11.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:59:11.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:59:11.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2023-11-02T08:59:11.943 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:59:11.943 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:59:11.944 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:59:11.944 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:59:11.944 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:59:11.944 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T08:59:11.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:59:11.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:59:11.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:59:11.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2023-11-02T08:59:11.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2023-11-02T08:59:11.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:59:11.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:59:11.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:59:11.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:59:11.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:59:11.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:59:11.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2023-11-02T08:59:11.948 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T08:59:11.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-11-02T08:59:11.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:59:11.950 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2023-11-02T08:59:11.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-11-02T08:59:11.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:59:11.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:59:11.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:59:11.966 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:59:11.963+0000 7f21e8e13500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:59:11.969 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:59:11.966+0000 7f21e8e13500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:59:11.970 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:59:11.967+0000 7f21e8e13500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:59:12.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-11-02T08:59:12.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:59:12.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T08:59:12.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:59:12.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:59:12.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:59:12.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:59:12.322 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:59:12.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:59:12.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:59:12.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:59:12.985 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:59:12.982+0000 7f21e8e13500 -1 Falling back to public interface 2023-11-02T08:59:13.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:59:13.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:59:13.734 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:59:13.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:59:13.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:59:13.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:59:14.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:59:15.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:59:15.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:59:15.115 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:59:15.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:59:15.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:59:15.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:59:15.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:59:16.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:59:16.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:59:16.518 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:59:16.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:59:16.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:59:16.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:59:16.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:59:17.593 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:59:17.590+0000 7f21e8e13500 -1 osd.0 50 log_to_monitors true 2023-11-02T08:59:17.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:59:17.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:59:17.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:59:17.931 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:59:17.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:59:17.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T08:59:18.351 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 59 up_thru 59 down_at 51 last_clean_interval [37,50) [v2:127.0.0.1:6802/186376,v1:127.0.0.1:6803/186376] [v2:127.0.0.1:6804/186376,v1:127.0.0.1:6805/186376] exists,up b30aad1c-508b-4a9d-871f-a21c604cf0a2 2023-11-02T08:59:18.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:59:18.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:59:18.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:59:18.353 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2023-11-02T08:59:18.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=186376 2023-11-02T08:59:18.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2023-11-02T08:59:18.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2023-11-02T08:59:18.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:59:18.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2023-11-02T08:59:18.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:59:18.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2023-11-02T08:59:18.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2023-11-02T08:59:18.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:59:18.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:59:18.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:59:18.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2023-11-02T08:59:18.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2023-11-02T08:59:18.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:59:18.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:59:18.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2023-11-02T08:59:18.359 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:59:18.359 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:59:18.360 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:59:18.360 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:59:18.360 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:59:18.360 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T08:59:18.360 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:59:18.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:59:18.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:59:18.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2023-11-02T08:59:18.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2023-11-02T08:59:18.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:59:18.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:59:18.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:59:18.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:59:18.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:59:18.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:59:18.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2023-11-02T08:59:18.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2023-11-02T08:59:18.363 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T08:59:18.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:59:18.364 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2023-11-02T08:59:18.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2023-11-02T08:59:18.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:59:18.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:59:18.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:59:18.380 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:59:18.376+0000 7fce48a0d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:59:18.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:59:18.396+0000 7fce48a0d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:59:18.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:59:18.397+0000 7fce48a0d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:59:18.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2023-11-02T08:59:18.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:59:18.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T08:59:18.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:59:18.762 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:59:18.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:59:18.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:59:18.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:59:18.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:59:18.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:59:19.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:59:19.414 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:59:19.411+0000 7fce48a0d500 -1 Falling back to public interface 2023-11-02T08:59:20.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:59:20.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:59:20.130 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:59:20.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:59:20.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:59:20.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:59:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:59:21.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:59:21.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:59:21.505 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:59:21.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:59:21.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:59:21.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:59:21.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:59:22.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:59:22.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:59:22.918 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:59:22.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:59:22.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:59:22.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:59:23.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:59:23.913 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:59:23.910+0000 7fce48a0d500 -1 osd.1 52 log_to_monitors true 2023-11-02T08:59:24.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:59:24.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:59:24.342 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:59:24.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:59:24.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:59:24.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T08:59:24.830 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 62 up_thru 62 down_at 53 last_clean_interval [40,52) [v2:127.0.0.1:6810/187345,v1:127.0.0.1:6811/187345] [v2:127.0.0.1:6812/187345,v1:127.0.0.1:6813/187345] exists,up d768ec0d-d275-4bc7-a822-c28b6a6e422a 2023-11-02T08:59:24.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:59:24.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:59:24.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:59:24.832 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2023-11-02T08:59:24.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=187345 2023-11-02T08:59:24.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2023-11-02T08:59:24.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2023-11-02T08:59:24.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:59:24.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2023-11-02T08:59:24.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:59:24.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2023-11-02T08:59:24.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2023-11-02T08:59:24.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:59:24.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:59:24.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:59:24.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2023-11-02T08:59:24.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2023-11-02T08:59:24.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:59:24.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:59:24.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2023-11-02T08:59:24.839 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:59:24.839 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:59:24.839 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:59:24.840 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:59:24.840 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:59:24.840 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T08:59:24.840 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:59:24.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:59:24.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:59:24.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2023-11-02T08:59:24.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2023-11-02T08:59:24.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:59:24.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:59:24.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:59:24.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:59:24.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:59:24.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:59:24.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2023-11-02T08:59:24.843 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T08:59:24.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2023-11-02T08:59:24.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:59:24.848 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2023-11-02T08:59:24.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2023-11-02T08:59:24.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:59:24.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:59:24.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:59:24.858 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:59:24.855+0000 7f1ffea1d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:59:24.860 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:59:24.857+0000 7f1ffea1d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:59:24.861 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:59:24.858+0000 7f1ffea1d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:59:25.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2023-11-02T08:59:25.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:59:25.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T08:59:25.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:59:25.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:59:25.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:59:25.221 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:59:25.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:59:25.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:59:25.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:59:25.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:59:25.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:59:25.873+0000 7f1ffea1d500 -1 Falling back to public interface 2023-11-02T08:59:26.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:59:26.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:59:26.620 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:59:26.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:59:26.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:59:26.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:59:26.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:59:28.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:59:28.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:59:28.001 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:59:28.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:59:28.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:59:28.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:59:28.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:59:29.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:59:29.412 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:59:29.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:59:29.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:59:29.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:59:29.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:59:29.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:59:30.623 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:59:30.621+0000 7f1ffea1d500 -1 osd.2 54 log_to_monitors true 2023-11-02T08:59:30.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:59:30.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:59:30.815 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T08:59:30.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T08:59:30.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:59:30.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T08:59:31.238 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 65 up_thru 53 down_at 55 last_clean_interval [44,54) [v2:127.0.0.1:6818/188314,v1:127.0.0.1:6819/188314] [v2:127.0.0.1:6820/188314,v1:127.0.0.1:6821/188314] exists,up 3b1ee458-46df-41df-a8cd-b9dff9a245c8 2023-11-02T08:59:31.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:59:31.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:59:31.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:59:31.239 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2023-11-02T08:59:31.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=188314 2023-11-02T08:59:31.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2023-11-02T08:59:31.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2023-11-02T08:59:31.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T08:59:31.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2023-11-02T08:59:31.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T08:59:31.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2023-11-02T08:59:31.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2023-11-02T08:59:31.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T08:59:31.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T08:59:31.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T08:59:31.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2023-11-02T08:59:31.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2023-11-02T08:59:31.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T08:59:31.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T08:59:31.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2023-11-02T08:59:31.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T08:59:31.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T08:59:31.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T08:59:31.259 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T08:59:31.259 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T08:59:31.260 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T08:59:31.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:59:31.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T08:59:31.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T08:59:31.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2023-11-02T08:59:31.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2023-11-02T08:59:31.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T08:59:31.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T08:59:31.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T08:59:31.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T08:59:31.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T08:59:31.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T08:59:31.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2023-11-02T08:59:31.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2023-11-02T08:59:31.263 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.3 2023-11-02T08:59:31.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T08:59:31.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2023-11-02T08:59:31.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2023-11-02T08:59:31.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T08:59:31.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T08:59:31.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T08:59:31.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:59:31.263+0000 7f543d411500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:59:31.272 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:59:31.266+0000 7f543d411500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:59:31.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:59:31.267+0000 7f543d411500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T08:59:31.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2023-11-02T08:59:31.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T08:59:31.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2023-11-02T08:59:31.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T08:59:31.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T08:59:31.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:59:31.886 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T08:59:31.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T08:59:31.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:59:31.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:59:32.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:59:32.032+0000 7f543d411500 -1 Falling back to public interface 2023-11-02T08:59:32.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:59:33.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:59:33.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:59:33.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T08:59:33.387 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T08:59:33.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:59:33.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:59:33.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:59:34.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:59:34.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:59:34.805 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T08:59:34.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T08:59:34.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:59:34.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:59:35.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T08:59:36.007 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T08:59:36.005+0000 7f543d411500 -1 osd.3 56 log_to_monitors true 2023-11-02T08:59:36.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T08:59:36.201 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T08:59:36.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T08:59:36.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T08:59:36.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T08:59:36.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T08:59:36.829 INFO:tasks.workunit.client.0.smithi116.stdout:osd.3 up in weight 1 up_from 69 up_thru 55 down_at 57 last_clean_interval [48,56) [v2:127.0.0.1:6826/189283,v1:127.0.0.1:6827/189283] [v2:127.0.0.1:6828/189283,v1:127.0.0.1:6829/189283] exists,up 22ca1871-a3a0-4021-b864-063cde3e60c0 2023-11-02T08:59:36.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T08:59:36.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T08:59:36.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T08:59:36.831 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2023-11-02T08:59:36.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=189283 2023-11-02T08:59:36.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2023-11-02T08:59:36.859 INFO:tasks.workunit.client.0.smithi116.stdout:hints = 1 2023-11-02T08:59:36.860 INFO:tasks.workunit.client.0.smithi116.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2023-11-02T08:59:36.860 INFO:tasks.workunit.client.0.smithi116.stdout:Object prefix: benchmark_data_smithi116_190207 2023-11-02T08:59:36.860 INFO:tasks.workunit.client.0.smithi116.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-11-02T08:59:36.861 INFO:tasks.workunit.client.0.smithi116.stdout: 0 0 0 0 0 0 - 0 2023-11-02T08:59:37.859 INFO:tasks.workunit.client.0.smithi116.stdout: 1 16 24 8 0.0312468 0.03125 0.828839 0.325902 2023-11-02T08:59:38.859 INFO:tasks.workunit.client.0.smithi116.stdout: 2 16 71 55 0.107411 0.183594 0.301675 0.561577 2023-11-02T08:59:39.860 INFO:tasks.workunit.client.0.smithi116.stdout: 3 16 114 98 0.127589 0.167969 0.299246 0.463338 2023-11-02T08:59:40.863 INFO:tasks.workunit.client.0.smithi116.stdout: 4 16 172 156 0.152324 0.226562 0.283479 0.401092 2023-11-02T08:59:41.860 INFO:tasks.workunit.client.0.smithi116.stdout: 5 16 221 205 0.160135 0.191406 0.284895 0.37313 2023-11-02T08:59:42.860 INFO:tasks.workunit.client.0.smithi116.stdout: 6 16 274 258 0.167947 0.207031 0.205184 0.362539 2023-11-02T08:59:43.862 INFO:tasks.workunit.client.0.smithi116.stdout: 7 16 327 311 0.173527 0.207031 0.223065 0.350566 2023-11-02T08:59:44.860 INFO:tasks.workunit.client.0.smithi116.stdout: 8 16 378 362 0.176735 0.199219 0.192632 0.348506 2023-11-02T08:59:45.860 INFO:tasks.workunit.client.0.smithi116.stdout: 9 16 428 412 0.178797 0.195312 0.207012 0.345185 2023-11-02T08:59:46.861 INFO:tasks.workunit.client.0.smithi116.stdout: 10 16 477 461 0.180056 0.191406 0.293672 0.339139 2023-11-02T08:59:47.861 INFO:tasks.workunit.client.0.smithi116.stdout: 11 16 532 516 0.183217 0.214844 0.317096 0.33599 2023-11-02T08:59:48.861 INFO:tasks.workunit.client.0.smithi116.stdout: 12 16 584 568 0.184873 0.203125 0.325018 0.332658 2023-11-02T08:59:49.862 INFO:tasks.workunit.client.0.smithi116.stdout: 13 16 634 618 0.185674 0.195312 0.343954 0.331163 2023-11-02T08:59:50.861 INFO:tasks.workunit.client.0.smithi116.stdout: 14 16 682 666 0.185802 0.1875 0.285697 0.332939 2023-11-02T08:59:51.861 INFO:tasks.workunit.client.0.smithi116.stdout: 15 16 733 717 0.186695 0.199219 0.214874 0.331002 2023-11-02T08:59:52.861 INFO:tasks.workunit.client.0.smithi116.stdout: 16 16 783 767 0.187233 0.195312 0.200133 0.329494 2023-11-02T08:59:53.861 INFO:tasks.workunit.client.0.smithi116.stdout: 17 16 840 824 0.189315 0.222656 0.309953 0.327349 2023-11-02T08:59:54.862 INFO:tasks.workunit.client.0.smithi116.stdout: 18 16 895 879 0.190732 0.214844 0.194443 0.324507 2023-11-02T08:59:55.863 INFO:tasks.workunit.client.0.smithi116.stdout: 19 16 945 929 0.190972 0.195312 0.292028 0.324685 2023-11-02T08:59:56.862 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T08:59:56.860039+0000 min lat: 0.145653 max lat: 1.39576 avg lat: 0.322678 2023-11-02T08:59:56.862 INFO:tasks.workunit.client.0.smithi116.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-11-02T08:59:56.862 INFO:tasks.workunit.client.0.smithi116.stdout: 20 16 998 982 0.191773 0.207031 0.310267 0.322678 2023-11-02T08:59:57.862 INFO:tasks.workunit.client.0.smithi116.stdout: 21 16 1052 1036 0.192685 0.210938 0.268381 0.321664 2023-11-02T08:59:58.862 INFO:tasks.workunit.client.0.smithi116.stdout: 22 16 1109 1093 0.194046 0.222656 0.246856 0.320176 2023-11-02T08:59:59.863 INFO:tasks.workunit.client.0.smithi116.stdout: 23 16 1162 1146 0.19461 0.207031 0.229279 0.319661 2023-11-02T09:00:00.862 INFO:tasks.workunit.client.0.smithi116.stdout: 24 16 1212 1196 0.194638 0.195312 0.318031 0.319857 2023-11-02T09:00:01.862 INFO:tasks.workunit.client.0.smithi116.stdout: 25 16 1262 1246 0.194664 0.195312 0.343687 0.318493 2023-11-02T09:00:02.862 INFO:tasks.workunit.client.0.smithi116.stdout: 26 16 1321 1305 0.196041 0.230469 0.275602 0.317682 2023-11-02T09:00:03.862 INFO:tasks.workunit.client.0.smithi116.stdout: 27 16 1369 1353 0.195724 0.1875 0.303776 0.317759 2023-11-02T09:00:04.862 INFO:tasks.workunit.client.0.smithi116.stdout: 28 16 1427 1411 0.196824 0.226562 0.308807 0.316042 2023-11-02T09:00:05.862 INFO:tasks.workunit.client.0.smithi116.stdout: 29 16 1468 1452 0.195559 0.160156 0.4959 0.317659 2023-11-02T09:00:06.863 INFO:tasks.workunit.client.0.smithi116.stdout: 30 16 1517 1501 0.19542 0.191406 0.344116 0.31737 2023-11-02T09:00:07.866 INFO:tasks.workunit.client.0.smithi116.stdout:Total time run: 30.1635 2023-11-02T09:00:07.866 INFO:tasks.workunit.client.0.smithi116.stdout:Total writes made: 1517 2023-11-02T09:00:07.867 INFO:tasks.workunit.client.0.smithi116.stdout:Write size: 4096 2023-11-02T09:00:07.867 INFO:tasks.workunit.client.0.smithi116.stdout:Object size: 4096 2023-11-02T09:00:07.867 INFO:tasks.workunit.client.0.smithi116.stdout:Bandwidth (MB/sec): 0.196455 2023-11-02T09:00:07.867 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev Bandwidth: 0.035006 2023-11-02T09:00:07.868 INFO:tasks.workunit.client.0.smithi116.stdout:Max bandwidth (MB/sec): 0.230469 2023-11-02T09:00:07.868 INFO:tasks.workunit.client.0.smithi116.stdout:Min bandwidth (MB/sec): 0.03125 2023-11-02T09:00:07.868 INFO:tasks.workunit.client.0.smithi116.stdout:Average IOPS: 50 2023-11-02T09:00:07.868 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev IOPS: 8.96154 2023-11-02T09:00:07.868 INFO:tasks.workunit.client.0.smithi116.stdout:Max IOPS: 59 2023-11-02T09:00:07.868 INFO:tasks.workunit.client.0.smithi116.stdout:Min IOPS: 8 2023-11-02T09:00:07.869 INFO:tasks.workunit.client.0.smithi116.stdout:Average Latency(s): 0.317241 2023-11-02T09:00:07.869 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev Latency(s): 0.110732 2023-11-02T09:00:07.869 INFO:tasks.workunit.client.0.smithi116.stdout:Max latency(s): 1.39576 2023-11-02T09:00:07.869 INFO:tasks.workunit.client.0.smithi116.stdout:Min latency(s): 0.145653 2023-11-02T09:00:07.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 186376 2023-11-02T09:00:07.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2023-11-02T09:00:08.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 186376 2023-11-02T09:00:08.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (186376) - No such process 2023-11-02T09:00:08.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2023-11-02T09:00:09.467 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 is already down. 2023-11-02T09:00:09.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 187345 2023-11-02T09:00:09.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2023-11-02T09:00:10.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 187345 2023-11-02T09:00:10.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (187345) - No such process 2023-11-02T09:00:10.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2023-11-02T09:00:11.049 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 is already down. 2023-11-02T09:00:11.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 188314 2023-11-02T09:00:11.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2023-11-02T09:00:12.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 188314 2023-11-02T09:00:12.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (188314) - No such process 2023-11-02T09:00:12.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2023-11-02T09:00:12.481 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 is already down. 2023-11-02T09:00:12.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 189283 2023-11-02T09:00:12.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2023-11-02T09:00:13.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 189283 2023-11-02T09:00:13.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (189283) - No such process 2023-11-02T09:00:13.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2023-11-02T09:00:14.015 INFO:tasks.workunit.client.0.smithi116.stderr:osd.3 is already down. 2023-11-02T09:00:14.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2023-11-02T09:00:15.808 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T09:00:15.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2023-11-02T09:00:16.314 INFO:tasks.workunit.client.0.smithi116.stderr:1024+0 records in 2023-11-02T09:00:16.315 INFO:tasks.workunit.client.0.smithi116.stderr:1024+0 records out 2023-11-02T09:00:16.315 INFO:tasks.workunit.client.0.smithi116.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.495848 s, 2.2 GB/s 2023-11-02T09:00:16.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2023-11-02T09:00:16.333 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T09:00:24.853 INFO:tasks.workunit.client.0.smithi116.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2023-11-02T09:00:24.853 INFO:tasks.workunit.client.0.smithi116.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2023-11-02T09:00:24.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2023-11-02T09:00:26.641 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T09:00:26.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2023-11-02T09:00:26.902 INFO:tasks.workunit.client.0.smithi116.stderr:512+0 records in 2023-11-02T09:00:26.903 INFO:tasks.workunit.client.0.smithi116.stderr:512+0 records out 2023-11-02T09:00:26.903 INFO:tasks.workunit.client.0.smithi116.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.253036 s, 2.1 GB/s 2023-11-02T09:00:26.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2023-11-02T09:00:26.921 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T09:00:32.191 INFO:tasks.workunit.client.0.smithi116.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2023-11-02T09:00:32.191 INFO:tasks.workunit.client.0.smithi116.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2023-11-02T09:00:32.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2023-11-02T09:00:32.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2023-11-02T09:00:34.033 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T09:00:34.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2023-11-02T09:00:35.827 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T09:00:35.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2023-11-02T09:00:36.337 INFO:tasks.workunit.client.0.smithi116.stderr:1024+0 records in 2023-11-02T09:00:36.338 INFO:tasks.workunit.client.0.smithi116.stderr:1024+0 records out 2023-11-02T09:00:36.338 INFO:tasks.workunit.client.0.smithi116.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.496352 s, 2.2 GB/s 2023-11-02T09:00:36.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2023-11-02T09:00:36.357 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T09:00:44.873 INFO:tasks.workunit.client.0.smithi116.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2023-11-02T09:00:44.873 INFO:tasks.workunit.client.0.smithi116.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2023-11-02T09:00:44.873 INFO:tasks.workunit.client.0.smithi116.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2023-11-02T09:00:44.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2023-11-02T09:00:44.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2023-11-02T09:00:46.258 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T09:00:46.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2023-11-02T09:00:46.284 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T09:00:47.797 INFO:tasks.workunit.client.0.smithi116.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2023-11-02T09:00:47.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2023-11-02T09:00:47.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2023-11-02T09:00:48.685 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T09:00:48.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2023-11-02T09:00:50.489 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T09:00:50.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2023-11-02T09:00:50.993 INFO:tasks.workunit.client.0.smithi116.stderr:1024+0 records in 2023-11-02T09:00:50.994 INFO:tasks.workunit.client.0.smithi116.stderr:1024+0 records out 2023-11-02T09:00:50.994 INFO:tasks.workunit.client.0.smithi116.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.494144 s, 2.2 GB/s 2023-11-02T09:00:50.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2023-11-02T09:00:51.013 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T09:00:51.014 INFO:tasks.workunit.client.0.smithi116.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2023-11-02T09:00:51.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2023-11-02T09:00:51.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2023-11-02T09:00:52.796 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T09:00:52.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2023-11-02T09:00:59.605 INFO:tasks.workunit.client.0.smithi116.stderr:1024+0 records in 2023-11-02T09:00:59.605 INFO:tasks.workunit.client.0.smithi116.stderr:1024+0 records out 2023-11-02T09:00:59.605 INFO:tasks.workunit.client.0.smithi116.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.68759 s, 161 MB/s 2023-11-02T09:00:59.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2023-11-02T09:00:59.626 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T09:01:01.639 INFO:tasks.workunit.client.0.smithi116.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2023-11-02T09:01:01.640 INFO:tasks.workunit.client.0.smithi116.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2023-11-02T09:01:01.640 INFO:tasks.workunit.client.0.smithi116.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2023-11-02T09:01:01.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2023-11-02T09:01:02.930 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T09:01:02.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2023-11-02T09:01:03.438 INFO:tasks.workunit.client.0.smithi116.stderr:1024+0 records in 2023-11-02T09:01:03.438 INFO:tasks.workunit.client.0.smithi116.stderr:1024+0 records out 2023-11-02T09:01:03.438 INFO:tasks.workunit.client.0.smithi116.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.499552 s, 2.1 GB/s 2023-11-02T09:01:03.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2023-11-02T09:01:03.457 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T09:01:11.223 INFO:tasks.workunit.client.0.smithi116.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2023-11-02T09:01:11.223 INFO:tasks.workunit.client.0.smithi116.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2023-11-02T09:01:11.224 INFO:tasks.workunit.client.0.smithi116.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2023-11-02T09:01:11.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2023-11-02T09:01:12.513 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T09:01:12.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2023-11-02T09:01:12.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:01:12.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T09:01:12.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-11-02T09:01:12.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T09:01:12.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2023-11-02T09:01:12.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2023-11-02T09:01:12.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:01:12.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:01:12.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:01:12.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2023-11-02T09:01:12.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2023-11-02T09:01:12.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T09:01:12.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T09:01:12.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2023-11-02T09:01:12.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T09:01:12.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:01:12.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:01:12.527 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:01:12.527 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:01:12.527 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:01:12.527 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T09:01:12.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T09:01:12.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:01:12.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2023-11-02T09:01:12.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2023-11-02T09:01:12.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:01:12.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:01:12.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:01:12.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:01:12.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T09:01:12.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T09:01:12.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2023-11-02T09:01:12.532 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T09:01:12.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-11-02T09:01:12.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:01:12.534 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2023-11-02T09:01:12.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-11-02T09:01:12.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T09:01:12.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T09:01:12.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T09:01:12.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:01:12.547+0000 7fc93b59b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:01:12.552 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:01:12.550+0000 7fc93b59b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:01:12.553 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:01:12.551+0000 7fc93b59b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:01:12.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-11-02T09:01:12.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:01:12.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T09:01:12.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:01:12.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:01:12.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:01:12.900 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:01:12.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:01:12.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:01:12.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:01:13.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:01:13.569 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:01:13.566+0000 7fc93b59b500 -1 Falling back to public interface 2023-11-02T09:01:14.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:01:14.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:01:14.330 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:01:14.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:01:14.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:01:14.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:01:14.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:01:15.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:01:15.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:01:15.709 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:01:15.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:01:15.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:01:15.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:01:16.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:01:17.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:01:17.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:01:17.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:01:17.090 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:01:17.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:01:17.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:01:17.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:01:18.070 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:01:18.067+0000 7fc93b59b500 -1 osd.0 71 log_to_monitors true 2023-11-02T09:01:18.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:01:18.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:01:18.476 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:01:18.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:01:18.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:01:18.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:01:18.949 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 81 up_thru 81 down_at 72 last_clean_interval [59,71) [v2:127.0.0.1:6802/196353,v1:127.0.0.1:6803/196353] [v2:127.0.0.1:6804/196353,v1:127.0.0.1:6805/196353] exists,up b30aad1c-508b-4a9d-871f-a21c604cf0a2 2023-11-02T09:01:18.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:01:18.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:01:18.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:01:18.956 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2023-11-02T09:01:18.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=196353 2023-11-02T09:01:18.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2023-11-02T09:01:18.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:01:18.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T09:01:18.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2023-11-02T09:01:18.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T09:01:18.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2023-11-02T09:01:18.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2023-11-02T09:01:18.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:01:18.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:01:18.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:01:18.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2023-11-02T09:01:18.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2023-11-02T09:01:18.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T09:01:18.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T09:01:18.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2023-11-02T09:01:18.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T09:01:18.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:01:18.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:01:18.963 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:01:18.963 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:01:18.963 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:01:18.964 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T09:01:18.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T09:01:18.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:01:18.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2023-11-02T09:01:18.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2023-11-02T09:01:18.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:01:18.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:01:18.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:01:18.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:01:18.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T09:01:18.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T09:01:18.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2023-11-02T09:01:18.969 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T09:01:18.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2023-11-02T09:01:18.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:01:18.970 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2023-11-02T09:01:18.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2023-11-02T09:01:18.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T09:01:18.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T09:01:18.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T09:01:18.977 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:01:18.974+0000 7f17436c5500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:01:19.009 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:01:19.006+0000 7f17436c5500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:01:19.010 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:01:19.007+0000 7f17436c5500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:01:19.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2023-11-02T09:01:19.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:01:19.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T09:01:19.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:01:19.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:01:19.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:01:19.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:01:19.337 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:01:19.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:01:19.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:01:19.535 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:01:19.533+0000 7f17436c5500 -1 Falling back to public interface 2023-11-02T09:01:19.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:01:20.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:01:20.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:01:20.784 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:01:20.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:01:20.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:01:20.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:01:21.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:01:22.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:01:22.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:01:22.166 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:01:22.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:01:22.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:01:22.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:01:22.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:01:22.484+0000 7f17436c5500 -1 osd.1 73 log_to_monitors true 2023-11-02T09:01:22.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:01:23.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:01:23.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:01:23.556 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:01:23.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:01:23.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:01:23.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:01:23.964 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 84 up_thru 84 down_at 74 last_clean_interval [62,73) [v2:127.0.0.1:6810/197322,v1:127.0.0.1:6811/197322] [v2:127.0.0.1:6812/197322,v1:127.0.0.1:6813/197322] exists,up d768ec0d-d275-4bc7-a822-c28b6a6e422a 2023-11-02T09:01:23.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:01:23.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:01:23.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:01:23.966 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2023-11-02T09:01:23.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=197322 2023-11-02T09:01:23.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2023-11-02T09:01:23.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:01:23.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T09:01:23.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2023-11-02T09:01:23.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T09:01:23.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2023-11-02T09:01:23.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2023-11-02T09:01:23.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:01:23.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:01:23.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:01:23.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2023-11-02T09:01:23.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2023-11-02T09:01:23.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T09:01:23.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T09:01:23.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2023-11-02T09:01:23.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T09:01:23.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:01:23.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:01:23.973 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:01:23.973 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:01:23.974 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:01:23.974 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T09:01:23.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T09:01:23.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:01:23.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2023-11-02T09:01:23.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2023-11-02T09:01:23.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:01:23.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:01:23.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:01:23.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:01:23.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T09:01:23.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T09:01:23.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2023-11-02T09:01:23.976 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T09:01:23.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2023-11-02T09:01:23.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:01:23.978 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2023-11-02T09:01:23.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2023-11-02T09:01:23.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T09:01:23.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T09:01:23.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T09:01:23.992 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:01:23.989+0000 7f8ebd416500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:01:23.994 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:01:23.991+0000 7f8ebd416500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:01:23.995 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:01:23.992+0000 7f8ebd416500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:01:24.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2023-11-02T09:01:24.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:01:24.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T09:01:24.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:01:24.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:01:24.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:01:24.377 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:01:24.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:01:24.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:01:24.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:01:24.760 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:01:24.757+0000 7f8ebd416500 -1 Falling back to public interface 2023-11-02T09:01:24.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:01:25.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:01:25.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:01:25.786 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:01:25.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:01:25.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:01:25.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:01:26.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:01:27.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:01:27.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:01:27.205 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:01:27.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:01:27.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:01:27.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:01:27.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:01:28.448 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:01:28.445+0000 7f8ebd416500 -1 osd.2 76 log_to_monitors true 2023-11-02T09:01:28.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:01:28.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:01:28.615 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:01:28.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:01:28.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:01:28.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:01:29.257 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 87 up_thru 87 down_at 77 last_clean_interval [65,76) [v2:127.0.0.1:6818/198256,v1:127.0.0.1:6819/198256] [v2:127.0.0.1:6820/198256,v1:127.0.0.1:6821/198256] exists,up 3b1ee458-46df-41df-a8cd-b9dff9a245c8 2023-11-02T09:01:29.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:01:29.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:01:29.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:01:29.258 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2023-11-02T09:01:29.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=198256 2023-11-02T09:01:29.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2023-11-02T09:01:29.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:01:29.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T09:01:29.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2023-11-02T09:01:29.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T09:01:29.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2023-11-02T09:01:29.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2023-11-02T09:01:29.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:01:29.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:01:29.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:01:29.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2023-11-02T09:01:29.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2023-11-02T09:01:29.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T09:01:29.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T09:01:29.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2023-11-02T09:01:29.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T09:01:29.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:01:29.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:01:29.290 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:01:29.291 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:01:29.291 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:01:29.291 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T09:01:29.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T09:01:29.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:01:29.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2023-11-02T09:01:29.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2023-11-02T09:01:29.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:01:29.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:01:29.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:01:29.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:01:29.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T09:01:29.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T09:01:29.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2023-11-02T09:01:29.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2023-11-02T09:01:29.294 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.3 2023-11-02T09:01:29.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:01:29.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2023-11-02T09:01:29.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2023-11-02T09:01:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T09:01:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T09:01:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T09:01:29.308 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:01:29.305+0000 7f470eb1a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:01:29.309 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:01:29.307+0000 7f470eb1a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:01:29.310 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:01:29.308+0000 7f470eb1a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:01:29.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2023-11-02T09:01:29.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:01:29.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2023-11-02T09:01:29.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:01:29.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:01:29.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:01:29.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:01:29.919 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:01:29.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:01:29.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T09:01:30.089 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:01:30.086+0000 7f470eb1a500 -1 Falling back to public interface 2023-11-02T09:01:30.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:01:31.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:01:31.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:01:31.294 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:01:31.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:01:31.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:01:31.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T09:01:31.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:01:32.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:01:32.707 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:01:32.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:01:32.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:01:32.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:01:32.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T09:01:33.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:01:33.550 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:01:33.547+0000 7f470eb1a500 -1 osd.3 78 log_to_monitors true 2023-11-02T09:01:34.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:01:34.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:01:34.117 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:01:34.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:01:34.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:01:34.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T09:01:34.665 INFO:tasks.workunit.client.0.smithi116.stdout:osd.3 up in weight 1 up_from 91 up_thru 91 down_at 79 last_clean_interval [69,78) [v2:127.0.0.1:6826/199195,v1:127.0.0.1:6827/199195] [v2:127.0.0.1:6828/199195,v1:127.0.0.1:6829/199195] exists,up 22ca1871-a3a0-4021-b864-063cde3e60c0 2023-11-02T09:01:34.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:01:34.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:01:34.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:01:34.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2023-11-02T09:01:34.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=199195 2023-11-02T09:01:34.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2023-11-02T09:01:34.711 INFO:tasks.workunit.client.0.smithi116.stdout:hints = 1 2023-11-02T09:01:34.712 INFO:tasks.workunit.client.0.smithi116.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2023-11-02T09:01:34.712 INFO:tasks.workunit.client.0.smithi116.stdout:Object prefix: benchmark_data_smithi116_200119 2023-11-02T09:01:34.712 INFO:tasks.workunit.client.0.smithi116.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-11-02T09:01:34.713 INFO:tasks.workunit.client.0.smithi116.stdout: 0 0 0 0 0 0 - 0 2023-11-02T09:01:35.712 INFO:tasks.workunit.client.0.smithi116.stdout: 1 16 90 74 0.289034 0.289062 0.175994 0.194344 2023-11-02T09:01:36.711 INFO:tasks.workunit.client.0.smithi116.stdout: 2 16 194 178 0.347615 0.40625 0.162328 0.167836 2023-11-02T09:01:37.712 INFO:tasks.workunit.client.0.smithi116.stdout: 3 16 301 285 0.371052 0.417969 0.138038 0.166821 2023-11-02T09:01:38.712 INFO:tasks.workunit.client.0.smithi116.stdout: 4 16 430 414 0.404254 0.503906 0.100213 0.152517 2023-11-02T09:01:39.712 INFO:tasks.workunit.client.0.smithi116.stdout: 5 16 547 531 0.4148 0.457031 0.0885095 0.149279 2023-11-02T09:01:40.712 INFO:tasks.workunit.client.0.smithi116.stdout: 6 16 675 659 0.428992 0.5 0.0790915 0.144978 2023-11-02T09:01:41.712 INFO:tasks.workunit.client.0.smithi116.stdout: 7 16 768 752 0.419597 0.363281 0.126974 0.146934 2023-11-02T09:01:42.712 INFO:tasks.workunit.client.0.smithi116.stdout: 8 16 889 873 0.426222 0.472656 0.13524 0.14558 2023-11-02T09:01:43.712 INFO:tasks.workunit.client.0.smithi116.stdout: 9 16 1002 986 0.427902 0.441406 0.125248 0.144792 2023-11-02T09:01:44.712 INFO:tasks.workunit.client.0.smithi116.stdout: 10 16 1111 1095 0.427686 0.425781 0.118283 0.144985 2023-11-02T09:01:45.713 INFO:tasks.workunit.client.0.smithi116.stdout: 11 16 1216 1200 0.426088 0.410156 0.102138 0.14547 2023-11-02T09:01:46.713 INFO:tasks.workunit.client.0.smithi116.stdout: 12 16 1353 1337 0.435173 0.535156 0.130628 0.142733 2023-11-02T09:01:47.713 INFO:tasks.workunit.client.0.smithi116.stdout: 13 16 1473 1457 0.437751 0.46875 0.0884521 0.141844 2023-11-02T09:01:48.715 INFO:tasks.workunit.client.0.smithi116.stdout: 14 16 1598 1582 0.441356 0.488281 0.134791 0.141004 2023-11-02T09:01:49.713 INFO:tasks.workunit.client.0.smithi116.stdout: 15 16 1701 1685 0.438752 0.402344 0.171093 0.141537 2023-11-02T09:01:50.713 INFO:tasks.workunit.client.0.smithi116.stdout: 16 16 1791 1775 0.4333 0.351562 0.174261 0.14328 2023-11-02T09:01:51.713 INFO:tasks.workunit.client.0.smithi116.stdout: 17 16 1904 1888 0.433773 0.441406 0.113094 0.143451 2023-11-02T09:01:52.714 INFO:tasks.workunit.client.0.smithi116.stdout: 18 16 2034 2018 0.437883 0.507812 0.12502 0.142056 2023-11-02T09:01:53.713 INFO:tasks.workunit.client.0.smithi116.stdout: 19 16 2145 2129 0.437654 0.433594 0.131787 0.142338 2023-11-02T09:01:54.714 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T09:01:54.712059+0000 min lat: 0.0589607 max lat: 0.627685 avg lat: 0.141835 2023-11-02T09:01:54.714 INFO:tasks.workunit.client.0.smithi116.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-11-02T09:01:54.714 INFO:tasks.workunit.client.0.smithi116.stdout: 20 16 2269 2253 0.439987 0.484375 0.125668 0.141835 2023-11-02T09:01:55.714 INFO:tasks.workunit.client.0.smithi116.stdout: 21 16 2349 2333 0.433915 0.3125 0.318541 0.143262 2023-11-02T09:01:56.714 INFO:tasks.workunit.client.0.smithi116.stdout: 22 16 2463 2447 0.434432 0.445312 0.121642 0.143485 2023-11-02T09:01:57.714 INFO:tasks.workunit.client.0.smithi116.stdout: 23 16 2566 2550 0.433035 0.402344 0.15079 0.143824 2023-11-02T09:01:58.714 INFO:tasks.workunit.client.0.smithi116.stdout: 24 16 2685 2669 0.434358 0.464844 0.125128 0.143379 2023-11-02T09:01:59.714 INFO:tasks.workunit.client.0.smithi116.stdout: 25 16 2797 2781 0.434483 0.4375 0.132835 0.143645 2023-11-02T09:02:00.714 INFO:tasks.workunit.client.0.smithi116.stdout: 26 16 2929 2913 0.437602 0.515625 0.12999 0.142739 2023-11-02T09:02:01.715 INFO:tasks.workunit.client.0.smithi116.stdout: 27 16 2995 2979 0.430941 0.257812 0.210112 0.144709 2023-11-02T09:02:02.714 INFO:tasks.workunit.client.0.smithi116.stdout: 28 15 3110 3095 0.431732 0.453125 0.148264 0.144489 2023-11-02T09:02:03.714 INFO:tasks.workunit.client.0.smithi116.stdout: 29 16 3221 3205 0.43166 0.429688 0.106591 0.144234 2023-11-02T09:02:04.715 INFO:tasks.workunit.client.0.smithi116.stdout: 30 16 3362 3346 0.435629 0.550781 0.115225 0.143182 2023-11-02T09:02:05.715 INFO:tasks.workunit.client.0.smithi116.stdout:Total time run: 30.0884 2023-11-02T09:02:05.716 INFO:tasks.workunit.client.0.smithi116.stdout:Total writes made: 3362 2023-11-02T09:02:05.716 INFO:tasks.workunit.client.0.smithi116.stdout:Write size: 4096 2023-11-02T09:02:05.716 INFO:tasks.workunit.client.0.smithi116.stdout:Object size: 4096 2023-11-02T09:02:05.716 INFO:tasks.workunit.client.0.smithi116.stdout:Bandwidth (MB/sec): 0.436474 2023-11-02T09:02:05.717 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev Bandwidth: 0.0689122 2023-11-02T09:02:05.717 INFO:tasks.workunit.client.0.smithi116.stdout:Max bandwidth (MB/sec): 0.550781 2023-11-02T09:02:05.717 INFO:tasks.workunit.client.0.smithi116.stdout:Min bandwidth (MB/sec): 0.257812 2023-11-02T09:02:05.717 INFO:tasks.workunit.client.0.smithi116.stdout:Average IOPS: 111 2023-11-02T09:02:05.718 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev IOPS: 17.6415 2023-11-02T09:02:05.718 INFO:tasks.workunit.client.0.smithi116.stdout:Max IOPS: 141 2023-11-02T09:02:05.718 INFO:tasks.workunit.client.0.smithi116.stdout:Min IOPS: 66 2023-11-02T09:02:05.718 INFO:tasks.workunit.client.0.smithi116.stdout:Average Latency(s): 0.143031 2023-11-02T09:02:05.718 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev Latency(s): 0.0468763 2023-11-02T09:02:05.719 INFO:tasks.workunit.client.0.smithi116.stdout:Max latency(s): 0.627685 2023-11-02T09:02:05.719 INFO:tasks.workunit.client.0.smithi116.stdout:Min latency(s): 0.0589607 2023-11-02T09:02:05.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2023-11-02T09:02:05.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T09:02:05.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T09:02:05.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T09:02:05.772 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-11-02T09:02:05.772 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T09:02:05.773 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T09:02:05.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T09:02:05.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T09:02:05.774 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T09:02:05.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-11-02T09:02:05.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T09:02:05.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T09:02:05.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T09:02:05.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T09:02:05.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T09:02:06.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T09:02:06.236 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T09:02:06.236 INFO:tasks.workunit.client.0.smithi116.stderr:2 2023-11-02T09:02:06.236 INFO:tasks.workunit.client.0.smithi116.stderr:3' 2023-11-02T09:02:06.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T09:02:06.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:02:06.237 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T09:02:06.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=347892350988 2023-11-02T09:02:06.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 347892350988 2023-11-02T09:02:06.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-347892350988' 2023-11-02T09:02:06.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:02:06.346 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T09:02:06.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=360777252875 2023-11-02T09:02:06.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 360777252875 2023-11-02T09:02:06.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-347892350988 1-360777252875' 2023-11-02T09:02:06.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:02:06.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T09:02:06.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=373662154762 2023-11-02T09:02:06.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 373662154762 2023-11-02T09:02:06.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-347892350988 1-360777252875 2-373662154762' 2023-11-02T09:02:06.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:02:06.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-11-02T09:02:06.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=390842023945 2023-11-02T09:02:06.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 390842023945 2023-11-02T09:02:06.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-347892350988 1-360777252875 2-373662154762 3-390842023945' 2023-11-02T09:02:06.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:02:06.679 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-347892350988 2023-11-02T09:02:06.679 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:02:06.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T09:02:06.680 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-347892350988 2023-11-02T09:02:06.680 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:02:06.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=347892350988 2023-11-02T09:02:06.681 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 347892350988 2023-11-02T09:02:06.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 347892350988' 2023-11-02T09:02:06.683 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:02:07.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 347892350988 -lt 347892350988 2023-11-02T09:02:07.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:02:07.045 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-360777252875 2023-11-02T09:02:07.045 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:02:07.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T09:02:07.047 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-360777252875 2023-11-02T09:02:07.047 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:02:07.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=360777252875 2023-11-02T09:02:07.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 360777252875' 2023-11-02T09:02:07.048 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 360777252875 2023-11-02T09:02:07.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T09:02:07.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 360777252875 -lt 360777252875 2023-11-02T09:02:07.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:02:07.418 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-373662154762 2023-11-02T09:02:07.418 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:02:07.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T09:02:07.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-373662154762 2023-11-02T09:02:07.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:02:07.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=373662154762 2023-11-02T09:02:07.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 373662154762' 2023-11-02T09:02:07.421 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 373662154762 2023-11-02T09:02:07.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T09:02:07.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 373662154762 -lt 373662154762 2023-11-02T09:02:07.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:02:07.837 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-390842023945 2023-11-02T09:02:07.838 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:02:07.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2023-11-02T09:02:07.839 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-390842023945 2023-11-02T09:02:07.840 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:02:07.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=390842023945 2023-11-02T09:02:07.841 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 390842023945 2023-11-02T09:02:07.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 390842023945' 2023-11-02T09:02:07.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2023-11-02T09:02:08.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 390842023945 -lt 390842023945 2023-11-02T09:02:08.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T09:02:08.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:02:08.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:02:08.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2023-11-02T09:02:08.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:02:08.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:02:08.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:02:08.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:02:08.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:02:08.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:02:08.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:02:09.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2023-11-02T09:02:09.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:02:09.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:02:09.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:02:09.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2023-11-02T09:02:09.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T09:02:09.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T09:02:09.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2023-11-02T09:02:09.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:02:09.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:02:09.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2023-11-02T09:02:09.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:02:09.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:02:09.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:02:09.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:02:09.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:02:09.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:02:09.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:02:09.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:02:09.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:02:09.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:02:09.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:02:09.758 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:02:09.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:02:09.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:02:09.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:02:09.760 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:02:09.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:02:09.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:02:09.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2023-11-02T09:02:10.163 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:02:10.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:02:10.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:02:10.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153754 2023-11-02T09:02:10.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:02:10.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:02:10.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2023-11-02T09:02:10.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2023-11-02T09:02:10.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:02:10.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2023-11-02T09:02:10.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:02:10.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:02:10.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2023-11-02T09:02:10.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:02:10.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:02:10.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:02:10.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:02:10.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:02:10.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:02:10.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:02:10.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:02:10.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:02:10.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:02:10.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:02:10.171 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:02:10.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:02:10.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:02:10.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:02:10.172 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:02:10.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:02:10.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:02:10.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2023-11-02T09:02:10.174 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:02:10.174 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:02:10.174 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:02:10.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153754 2023-11-02T09:02:10.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:02:10.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:02:10.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2023-11-02T09:02:10.176 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T09:02:10.176 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:02:10.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:02:10.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.153754 2023-11-02T09:02:10.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T09:02:10.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-11-02T09:02:10.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T09:02:10.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2023-11-02T09:02:10.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore2 td/osd-bluefs-volume-ops 2023-11-02T09:02:10.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:341: TEST_bluestore2: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:02:10.179 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: ulimit -n 2023-11-02T09:02:10.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: local flimit=4096 2023-11-02T09:02:10.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:344: TEST_bluestore2: '[' 4096 -lt 1536 ']' 2023-11-02T09:02:10.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: export CEPH_MON=127.0.0.1:7146 2023-11-02T09:02:10.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: CEPH_MON=127.0.0.1:7146 2023-11-02T09:02:10.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:348: TEST_bluestore2: export CEPH_ARGS 2023-11-02T09:02:10.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: uuidgen 2023-11-02T09:02:10.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: CEPH_ARGS+='--fsid=3aaadb38-61d9-40f4-8cb1-362d1e219519 --auth-supported=none ' 2023-11-02T09:02:10.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:350: TEST_bluestore2: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2023-11-02T09:02:10.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:351: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2023-11-02T09:02:10.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:352: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_create=true ' 2023-11-02T09:02:10.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:353: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2023-11-02T09:02:10.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:354: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2023-11-02T09:02:10.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:355: TEST_bluestore2: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2023-11-02T09:02:10.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:356: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_size=1 ' 2023-11-02T09:02:10.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:357: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2023-11-02T09:02:10.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:358: TEST_bluestore2: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2023-11-02T09:02:10.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:360: TEST_bluestore2: run_mon td/osd-bluefs-volume-ops a 2023-11-02T09:02:10.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:02:10.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T09:02:10.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T09:02:10.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T09:02:10.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2023-11-02T09:02:10.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2023-11-02T09:02:10.561 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T09:02:10.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:02:10.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:02:10.562 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:02:10.562 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:02:10.562 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:02:10.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T09:02:10.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T09:02:10.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T09:02:10.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T09:02:10.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:02:10.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:02:10.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T09:02:10.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T09:02:10.756 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:02:10.756 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:02:10.757 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:02:10.757 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:02:10.757 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:02:10.757 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:02:10.757 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153754/ceph-mon.a.asok 2023-11-02T09:02:10.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:02:10.758 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.153754/ceph-mon.a.asok config get fsid 2023-11-02T09:02:10.834 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T09:02:10.834 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:02:10.835 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:02:10.835 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T09:02:10.835 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T09:02:10.836 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:02:10.836 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:02:10.836 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:02:10.837 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:02:10.837 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:02:10.837 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:02:10.837 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153754/ceph-mon.a.asok 2023-11-02T09:02:10.837 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:02:10.838 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.153754/ceph-mon.a.asok config get mon_host 2023-11-02T09:02:10.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:361: TEST_bluestore2: run_mgr td/osd-bluefs-volume-ops x 2023-11-02T09:02:10.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:02:10.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T09:02:10.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T09:02:10.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T09:02:10.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2023-11-02T09:02:10.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T09:02:11.758 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T09:02:11.758 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:02:11.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:02:11.759 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:02:11.759 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:02:11.759 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:02:11.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T09:02:11.760 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:02:11.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:02:11.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:362: TEST_bluestore2: run_osd td/osd-bluefs-volume-ops 0 2023-11-02T09:02:11.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:02:11.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:02:11.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T09:02:11.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:02:11.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2023-11-02T09:02:11.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=3aaadb38-61d9-40f4-8cb1-362d1e219519 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2023-11-02T09:02:11.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:02:11.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:02:11.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:02:11.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2023-11-02T09:02:11.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2023-11-02T09:02:11.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:02:11.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:02:11.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2023-11-02T09:02:11.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:02:11.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:02:11.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:02:11.795 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:02:11.795 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:02:11.795 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:02:11.796 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T09:02:11.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T09:02:11.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:02:11.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:02:11.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:02:11.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2023-11-02T09:02:11.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2023-11-02T09:02:11.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:02:11.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:02:11.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:02:11.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:02:11.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:02:11.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:02:11.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2023-11-02T09:02:11.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:02:11.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f243181d-da36-48d1-bf0a-7599dd11c96d 2023-11-02T09:02:11.799 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 f243181d-da36-48d1-bf0a-7599dd11c96d 2023-11-02T09:02:11.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f243181d-da36-48d1-bf0a-7599dd11c96d' 2023-11-02T09:02:11.800 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:02:11.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTZUNlGew+MBAAZLGgFF4Kdfd4UDAivBW8QA== 2023-11-02T09:02:11.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTZUNlGew+MBAAZLGgFF4Kdfd4UDAivBW8QA=="}' 2023-11-02T09:02:11.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f243181d-da36-48d1-bf0a-7599dd11c96d -i td/osd-bluefs-volume-ops/0/new.json 2023-11-02T09:02:12.096 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:02:12.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2023-11-02T09:02:12.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=3aaadb38-61d9-40f4-8cb1-362d1e219519 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTZUNlGew+MBAAZLGgFF4Kdfd4UDAivBW8QA== --osd-uuid f243181d-da36-48d1-bf0a-7599dd11c96d 2023-11-02T09:02:12.139 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:02:12.135+0000 7f13cae38500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:02:12.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:02:12.144+0000 7f13cae38500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:02:12.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:02:12.145+0000 7f13cae38500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:02:12.148 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:02:12.145+0000 7f13cae38500 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2023-11-02T09:02:12.148 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:02:12.145+0000 7f13cae38500 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2023-11-02T09:02:12.149 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:02:12.145+0000 7f13cae38500 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2023-11-02T09:02:12.186 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:02:12.183+0000 7f13cae38500 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2023-11-02T09:02:19.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2023-11-02T09:02:19.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:02:19.642 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T09:02:19.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T09:02:19.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:02:20.161 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T09:02:20.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T09:02:20.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=3aaadb38-61d9-40f4-8cb1-362d1e219519 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:02:20.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:02:20.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:02:20.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:02:20.193 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:02:20.190+0000 7fb6dd307500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:02:20.200 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:02:20.197+0000 7fb6dd307500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:02:20.201 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:02:20.198+0000 7fb6dd307500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:02:20.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T09:02:20.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:02:20.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T09:02:20.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:02:20.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:02:20.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:02:20.623 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:02:20.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:02:20.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:02:20.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:02:20.987 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:02:20.984+0000 7fb6dd307500 -1 Falling back to public interface 2023-11-02T09:02:21.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:02:22.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:02:22.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:02:22.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:02:22.033 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:02:22.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:02:22.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:02:22.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:02:23.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:02:23.415 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:02:23.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:02:23.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:02:23.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:02:23.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:02:23.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:02:24.737 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:02:24.734+0000 7fb6dd307500 -1 osd.0 0 log_to_monitors true 2023-11-02T09:02:24.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:02:24.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:02:24.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:02:24.835 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:02:24.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:02:24.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:02:25.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:02:26.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:02:26.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:02:26.421 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:02:26.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:02:26.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:02:26.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:02:27.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:02:28.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:02:28.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:02:28.970 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:02:28.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:02:28.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:02:28.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:02:29.386 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/201989,v1:127.0.0.1:6803/201989] [v2:127.0.0.1:6804/201989,v1:127.0.0.1:6805/201989] exists,up f243181d-da36-48d1-bf0a-7599dd11c96d 2023-11-02T09:02:29.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:02:29.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:02:29.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:02:29.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2023-11-02T09:02:29.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: osd_pid0=201989 2023-11-02T09:02:29.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:365: TEST_bluestore2: sleep 5 2023-11-02T09:02:34.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:366: TEST_bluestore2: create_pool foo 16 2023-11-02T09:02:34.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2023-11-02T09:02:34.922 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'foo' already exists 2023-11-02T09:02:34.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:02:35.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:368: TEST_bluestore2: retry = 0 2023-11-02T09:02:35.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 368: retry: command not found 2023-11-02T09:02:35.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:369: TEST_bluestore2: [[ '' -le 5 ]] 2023-11-02T09:02:35.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:371: TEST_bluestore2: timeout 60 rados bench -p foo 10 write --write-omap --no-cleanup 2023-11-02T09:02:35.973 INFO:tasks.workunit.client.0.smithi116.stdout:hints = 1 2023-11-02T09:02:35.974 INFO:tasks.workunit.client.0.smithi116.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects 2023-11-02T09:02:35.974 INFO:tasks.workunit.client.0.smithi116.stdout:Object prefix: benchmark_data_smithi116_203245 2023-11-02T09:02:36.004 INFO:tasks.workunit.client.0.smithi116.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-11-02T09:02:36.004 INFO:tasks.workunit.client.0.smithi116.stdout: 0 0 0 0 0 0 - 0 2023-11-02T09:02:37.004 INFO:tasks.workunit.client.0.smithi116.stdout: 1 16 33 17 67.9947 68 0.742868 0.752483 2023-11-02T09:02:38.004 INFO:tasks.workunit.client.0.smithi116.stdout: 2 16 49 33 65.9925 64 0.751515 0.774596 2023-11-02T09:02:39.004 INFO:tasks.workunit.client.0.smithi116.stdout: 3 16 65 49 65.3248 64 1.02783 0.857263 2023-11-02T09:02:40.004 INFO:tasks.workunit.client.0.smithi116.stdout: 4 16 80 64 63.9919 60 1.11882 0.917885 2023-11-02T09:02:41.004 INFO:tasks.workunit.client.0.smithi116.stdout: 5 16 81 65 51.9938 4 1.44401 0.925979 2023-11-02T09:02:42.004 INFO:tasks.workunit.client.0.smithi116.stdout: 6 16 81 65 43.3281 0 - 0.925979 2023-11-02T09:02:43.005 INFO:tasks.workunit.client.0.smithi116.stdout: 7 16 97 81 46.2801 32 2.36245 1.23286 2023-11-02T09:02:44.005 INFO:tasks.workunit.client.0.smithi116.stdout: 8 16 97 81 40.4952 0 - 1.23286 2023-11-02T09:02:45.005 INFO:tasks.workunit.client.0.smithi116.stdout: 9 16 128 112 49.772 62 0.776565 1.2767 2023-11-02T09:02:46.005 INFO:tasks.workunit.client.0.smithi116.stdout: 10 16 145 129 51.5937 68 0.775974 1.20278 2023-11-02T09:02:47.005 INFO:tasks.workunit.client.0.smithi116.stdout:Total time run: 10.9915 2023-11-02T09:02:47.006 INFO:tasks.workunit.client.0.smithi116.stdout:Total writes made: 145 2023-11-02T09:02:47.006 INFO:tasks.workunit.client.0.smithi116.stdout:Write size: 4194304 2023-11-02T09:02:47.006 INFO:tasks.workunit.client.0.smithi116.stdout:Object size: 4194304 2023-11-02T09:02:47.006 INFO:tasks.workunit.client.0.smithi116.stdout:Bandwidth (MB/sec): 52.7678 2023-11-02T09:02:47.006 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev Bandwidth: 30.0289 2023-11-02T09:02:47.007 INFO:tasks.workunit.client.0.smithi116.stdout:Max bandwidth (MB/sec): 68 2023-11-02T09:02:47.007 INFO:tasks.workunit.client.0.smithi116.stdout:Min bandwidth (MB/sec): 0 2023-11-02T09:02:47.007 INFO:tasks.workunit.client.0.smithi116.stdout:Average IOPS: 13 2023-11-02T09:02:47.007 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev IOPS: 7.47217 2023-11-02T09:02:47.007 INFO:tasks.workunit.client.0.smithi116.stdout:Max IOPS: 17 2023-11-02T09:02:47.008 INFO:tasks.workunit.client.0.smithi116.stdout:Min IOPS: 0 2023-11-02T09:02:47.008 INFO:tasks.workunit.client.0.smithi116.stdout:Average Latency(s): 1.19906 2023-11-02T09:02:47.008 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev Latency(s): 0.585886 2023-11-02T09:02:47.008 INFO:tasks.workunit.client.0.smithi116.stdout:Max latency(s): 2.4937 2023-11-02T09:02:47.008 INFO:tasks.workunit.client.0.smithi116.stdout:Min latency(s): 0.189263 2023-11-02T09:02:47.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:374: TEST_bluestore2: sleep 10 2023-11-02T09:02:57.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2023-11-02T09:02:57.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: jq .bluefs.db_used_bytes 2023-11-02T09:02:57.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: db_used=1069547520 2023-11-02T09:02:57.270 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2023-11-02T09:02:57.270 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: jq .bluefs.slow_used_bytes 2023-11-02T09:02:57.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: spilled_over=446496768 2023-11-02T09:02:57.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:378: TEST_bluestore2: (( retry+=1 )) 2023-11-02T09:02:57.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: test 446496768 -eq 0 2023-11-02T09:02:57.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: break 2023-11-02T09:02:57.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:381: TEST_bluestore2: test 446496768 -gt 0 2023-11-02T09:02:57.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 201989 2023-11-02T09:02:57.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: sleep 1 2023-11-02T09:02:58.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 201989 2023-11-02T09:02:58.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 383: kill: (201989) - No such process 2023-11-02T09:02:58.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:384: TEST_bluestore2: ceph osd down 0 2023-11-02T09:02:58.794 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 is already down. 2023-11-02T09:02:58.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:386: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/block --command bluefs-bdev-migrate 2023-11-02T09:02:58.827 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T09:03:07.593 INFO:tasks.workunit.client.0.smithi116.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2023-11-02T09:03:07.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:391: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command bluefs-bdev-sizes 2023-11-02T09:03:07.629 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T09:03:08.892 INFO:tasks.workunit.client.0.smithi116.stdout:1 : device size 0x100000000 : using 0x5a7e9000(1.4 GiB) 2023-11-02T09:03:08.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:394: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command fsck 2023-11-02T09:03:10.690 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T09:03:10.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:397: TEST_bluestore2: activate_osd td/osd-bluefs-volume-ops 0 2023-11-02T09:03:10.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:03:10.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T09:03:10.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-11-02T09:03:10.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T09:03:10.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2023-11-02T09:03:10.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=3aaadb38-61d9-40f4-8cb1-362d1e219519 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2023-11-02T09:03:10.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:03:10.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:03:10.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:03:10.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2023-11-02T09:03:10.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2023-11-02T09:03:10.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T09:03:10.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T09:03:10.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2023-11-02T09:03:10.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T09:03:10.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:03:10.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:03:10.722 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:03:10.723 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:03:10.723 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:03:10.723 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T09:03:10.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T09:03:10.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:03:10.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2023-11-02T09:03:10.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2023-11-02T09:03:10.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:03:10.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:03:10.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:03:10.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:03:10.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T09:03:10.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T09:03:10.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2023-11-02T09:03:10.726 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T09:03:10.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-11-02T09:03:10.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=3aaadb38-61d9-40f4-8cb1-362d1e219519 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:03:10.729 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2023-11-02T09:03:10.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-11-02T09:03:10.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T09:03:10.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T09:03:10.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T09:03:10.749 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:03:10.746+0000 7f79434da500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:03:10.751 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:03:10.748+0000 7f79434da500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:03:10.752 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:03:10.749+0000 7f79434da500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:03:11.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-11-02T09:03:11.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:03:11.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T09:03:11.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:03:11.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:03:11.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:03:11.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:03:11.122 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:03:11.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:03:11.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:03:11.268 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:03:11.265+0000 7f79434da500 -1 Falling back to public interface 2023-11-02T09:03:11.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:03:12.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:03:12.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:03:12.495 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:03:12.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:03:12.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:03:12.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:03:12.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:03:13.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:03:13.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:03:13.917 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:03:13.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:03:13.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:03:13.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:03:14.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:03:15.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:03:15.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:03:15.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:03:15.305 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:03:15.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:03:15.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:03:15.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:03:16.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:03:16.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:03:16.709 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:03:16.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:03:16.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:03:16.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:03:17.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:03:18.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:03:18.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:03:18.095 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:03:18.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:03:18.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:03:18.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:03:18.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:03:19.132 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:03:19.129+0000 7f79434da500 -1 osd.0 10 log_to_monitors true 2023-11-02T09:03:19.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:03:19.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:03:19.484 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:03:19.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:03:19.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:03:19.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:03:21.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:03:22.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:03:22.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:03:22.151 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:03:22.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:03:22.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:03:22.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:03:22.402 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:03:22.399+0000 7f79434da500 -1 bdev(0x55f60f3c0e00 td/osd-bluefs-volume-ops/0/block) aio_submit retries 6 2023-11-02T09:03:22.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:03:23.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:03:23.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:03:23.620 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:03:23.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:03:23.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:03:23.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:03:24.070 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 13 up_thru 13 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/204470,v1:127.0.0.1:6803/204470] [v2:127.0.0.1:6804/204470,v1:127.0.0.1:6805/204470] exists,up f243181d-da36-48d1-bf0a-7599dd11c96d 2023-11-02T09:03:24.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:03:24.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:03:24.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:03:24.071 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2023-11-02T09:03:24.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: osd_pid0=204470 2023-11-02T09:03:24.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:400: TEST_bluestore2: wait_for_clean 2023-11-02T09:03:24.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T09:03:24.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T09:03:24.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T09:03:24.074 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-11-02T09:03:24.074 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T09:03:24.074 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T09:03:24.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T09:03:24.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T09:03:24.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T09:03:24.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-11-02T09:03:24.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T09:03:24.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T09:03:24.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T09:03:24.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T09:03:24.155 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T09:03:24.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2023-11-02T09:03:24.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T09:03:24.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:03:24.548 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T09:03:24.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=55834574852 2023-11-02T09:03:24.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 55834574852 2023-11-02T09:03:24.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-55834574852' 2023-11-02T09:03:24.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:03:24.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-55834574852 2023-11-02T09:03:24.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:03:24.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T09:03:24.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-55834574852 2023-11-02T09:03:24.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:03:24.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=55834574852 2023-11-02T09:03:24.668 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 55834574852 2023-11-02T09:03:24.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2023-11-02T09:03:24.668 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:03:25.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 1 -lt 55834574852 2023-11-02T09:03:25.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T09:03:26.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T09:03:26.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:03:26.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 55834574852 -lt 55834574852 2023-11-02T09:03:26.528 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T09:03:26.529 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:03:26.529 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:03:27.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2023-11-02T09:03:27.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:03:27.033 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:03:27.033 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:03:27.033 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:03:27.033 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:03:27.034 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:03:27.034 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:03:27.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2023-11-02T09:03:27.442 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:03:27.442 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:03:27.442 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:03:27.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2023-11-02T09:03:27.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T09:03:27.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T09:03:27.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2023-11-02T09:03:27.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:03:27.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:03:27.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2023-11-02T09:03:27.966 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:03:27.967 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:03:27.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:03:27.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:03:27.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:03:28.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:03:28.071 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:03:28.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:03:28.072 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:03:28.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:03:28.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:03:28.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:03:28.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:03:28.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:03:28.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:03:28.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:03:28.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:03:28.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:03:28.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2023-11-02T09:03:28.214 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:03:28.215 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:03:28.215 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:03:28.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153754 2023-11-02T09:03:28.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:03:28.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:03:28.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2023-11-02T09:03:28.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2023-11-02T09:03:28.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:03:28.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2023-11-02T09:03:28.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:03:28.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:03:28.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2023-11-02T09:03:28.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:03:28.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:03:28.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:03:28.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:03:28.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:03:28.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:03:28.223 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:03:28.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:03:28.224 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:03:28.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:03:28.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:03:28.224 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:03:28.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:03:28.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:03:28.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:03:28.225 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:03:28.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:03:28.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:03:28.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2023-11-02T09:03:28.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:03:28.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:03:28.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:03:28.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153754 2023-11-02T09:03:28.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:03:28.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:03:28.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2023-11-02T09:03:28.230 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T09:03:28.231 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:03:28.231 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:03:28.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.153754 2023-11-02T09:03:28.232 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T09:03:28.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-11-02T09:03:28.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T09:03:28.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2023-11-02T09:03:28.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore_expand td/osd-bluefs-volume-ops 2023-11-02T09:03:28.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:404: TEST_bluestore_expand: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:03:28.233 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: ulimit -n 2023-11-02T09:03:28.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: local flimit=4096 2023-11-02T09:03:28.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:407: TEST_bluestore_expand: '[' 4096 -lt 1536 ']' 2023-11-02T09:03:28.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: export CEPH_MON=127.0.0.1:7146 2023-11-02T09:03:28.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: CEPH_MON=127.0.0.1:7146 2023-11-02T09:03:28.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:411: TEST_bluestore_expand: export CEPH_ARGS 2023-11-02T09:03:28.237 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: uuidgen 2023-11-02T09:03:28.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: CEPH_ARGS+='--fsid=c6749193-4650-4de3-9cb2-c42bf836b301 --auth-supported=none ' 2023-11-02T09:03:28.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:413: TEST_bluestore_expand: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2023-11-02T09:03:28.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:414: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2023-11-02T09:03:28.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:415: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_create=true ' 2023-11-02T09:03:28.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:416: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2023-11-02T09:03:28.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:417: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2023-11-02T09:03:28.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:418: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2023-11-02T09:03:28.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:419: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_size=1 ' 2023-11-02T09:03:28.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:420: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2023-11-02T09:03:28.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:421: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2023-11-02T09:03:28.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:423: TEST_bluestore_expand: run_mon td/osd-bluefs-volume-ops a 2023-11-02T09:03:28.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:03:28.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T09:03:28.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T09:03:28.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T09:03:28.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2023-11-02T09:03:28.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2023-11-02T09:03:28.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T09:03:28.609 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:03:28.609 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:03:28.609 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:03:28.609 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:03:28.610 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:03:28.610 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T09:03:28.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T09:03:28.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T09:03:28.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T09:03:28.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:03:28.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:03:28.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T09:03:28.812 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T09:03:28.812 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:03:28.812 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:03:28.812 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:03:28.812 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:03:28.813 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:03:28.813 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:03:28.813 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153754/ceph-mon.a.asok 2023-11-02T09:03:28.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:03:28.814 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.153754/ceph-mon.a.asok config get fsid 2023-11-02T09:03:28.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T09:03:28.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:03:28.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:03:28.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T09:03:28.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T09:03:28.892 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:03:28.893 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:03:28.893 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:03:28.893 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:03:28.894 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:03:28.894 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:03:28.894 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153754/ceph-mon.a.asok 2023-11-02T09:03:28.896 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:03:28.897 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.153754/ceph-mon.a.asok config get mon_host 2023-11-02T09:03:28.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:424: TEST_bluestore_expand: run_mgr td/osd-bluefs-volume-ops x 2023-11-02T09:03:28.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:03:28.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T09:03:28.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T09:03:28.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T09:03:28.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2023-11-02T09:03:28.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T09:03:29.831 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T09:03:29.831 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:03:29.831 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:03:29.831 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:03:29.832 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:03:29.832 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:03:29.832 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T09:03:29.833 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:03:29.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:03:29.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:425: TEST_bluestore_expand: run_osd td/osd-bluefs-volume-ops 0 2023-11-02T09:03:29.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:03:29.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:03:29.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T09:03:29.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:03:29.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2023-11-02T09:03:29.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=3aaadb38-61d9-40f4-8cb1-362d1e219519 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=c6749193-4650-4de3-9cb2-c42bf836b301 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2023-11-02T09:03:29.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:03:29.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:03:29.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:03:29.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2023-11-02T09:03:29.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2023-11-02T09:03:29.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:03:29.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:03:29.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2023-11-02T09:03:29.870 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:03:29.870 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:03:29.870 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:03:29.871 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:03:29.871 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:03:29.871 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:03:29.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T09:03:29.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T09:03:29.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:03:29.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:03:29.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:03:29.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2023-11-02T09:03:29.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2023-11-02T09:03:29.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:03:29.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:03:29.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:03:29.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:03:29.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:03:29.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:03:29.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2023-11-02T09:03:29.874 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:03:29.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f96ec9f8-f335-4060-826e-02fd5f4b07ba 2023-11-02T09:03:29.875 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 f96ec9f8-f335-4060-826e-02fd5f4b07ba 2023-11-02T09:03:29.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f96ec9f8-f335-4060-826e-02fd5f4b07ba' 2023-11-02T09:03:29.876 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:03:29.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDhZUNl9LwINRAArPWXVarj/lc7TMUfssw59A== 2023-11-02T09:03:29.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDhZUNl9LwINRAArPWXVarj/lc7TMUfssw59A=="}' 2023-11-02T09:03:29.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f96ec9f8-f335-4060-826e-02fd5f4b07ba -i td/osd-bluefs-volume-ops/0/new.json 2023-11-02T09:03:30.215 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:03:30.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2023-11-02T09:03:30.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=3aaadb38-61d9-40f4-8cb1-362d1e219519 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=c6749193-4650-4de3-9cb2-c42bf836b301 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhZUNl9LwINRAArPWXVarj/lc7TMUfssw59A== --osd-uuid f96ec9f8-f335-4060-826e-02fd5f4b07ba 2023-11-02T09:03:30.260 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:03:30.257+0000 7fb976d37500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:03:30.262 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:03:30.259+0000 7fb976d37500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:03:30.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:03:30.260+0000 7fb976d37500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:03:30.263 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:03:30.261+0000 7fb976d37500 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2023-11-02T09:03:30.264 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:03:30.261+0000 7fb976d37500 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2023-11-02T09:03:30.264 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:03:30.261+0000 7fb976d37500 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2023-11-02T09:03:30.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:03:30.333+0000 7fb976d37500 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2023-11-02T09:03:37.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2023-11-02T09:03:37.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:03:37.292 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T09:03:37.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T09:03:37.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:03:37.820 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T09:03:37.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T09:03:37.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:03:37.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=3aaadb38-61d9-40f4-8cb1-362d1e219519 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=c6749193-4650-4de3-9cb2-c42bf836b301 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:03:37.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:03:37.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:03:37.856 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:03:37.853+0000 7fb89100e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:03:37.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:03:37.873+0000 7fb89100e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:03:37.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:03:37.874+0000 7fb89100e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:03:38.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T09:03:38.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:03:38.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T09:03:38.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:03:38.235 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:03:38.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:03:38.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:03:38.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:03:38.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:03:38.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:03:38.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:03:38.664 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:03:38.662+0000 7fb89100e500 -1 Falling back to public interface 2023-11-02T09:03:39.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:03:39.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:03:39.650 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:03:39.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:03:39.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:03:39.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:03:40.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:03:41.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:03:41.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:03:41.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:03:41.068 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:03:41.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:03:41.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:03:41.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:03:42.411 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:03:42.407+0000 7fb89100e500 -1 osd.0 0 log_to_monitors true 2023-11-02T09:03:42.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:03:42.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:03:42.485 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:03:42.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:03:42.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:03:42.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:03:43.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:03:44.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:03:44.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:03:44.103 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:03:44.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:03:44.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:03:44.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:03:45.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:03:46.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:03:46.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:03:46.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:03:46.478 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:03:46.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:03:46.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:03:46.881 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/207217,v1:127.0.0.1:6803/207217] [v2:127.0.0.1:6804/207217,v1:127.0.0.1:6805/207217] exists,up f96ec9f8-f335-4060-826e-02fd5f4b07ba 2023-11-02T09:03:46.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:03:46.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:03:46.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:03:46.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2023-11-02T09:03:46.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: osd_pid0=207217 2023-11-02T09:03:46.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:428: TEST_bluestore_expand: sleep 5 2023-11-02T09:03:51.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:429: TEST_bluestore_expand: create_pool foo 16 2023-11-02T09:03:51.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2023-11-02T09:03:52.499 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'foo' already exists 2023-11-02T09:03:52.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:03:53.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:432: TEST_bluestore_expand: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2023-11-02T09:03:53.555 INFO:tasks.workunit.client.0.smithi116.stdout:hints = 1 2023-11-02T09:03:53.555 INFO:tasks.workunit.client.0.smithi116.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2023-11-02T09:03:53.555 INFO:tasks.workunit.client.0.smithi116.stdout:Object prefix: benchmark_data_smithi116_208218 2023-11-02T09:03:53.556 INFO:tasks.workunit.client.0.smithi116.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-11-02T09:03:53.556 INFO:tasks.workunit.client.0.smithi116.stdout: 0 0 0 0 0 0 - 0 2023-11-02T09:03:54.552 INFO:tasks.workunit.client.0.smithi116.stdout: 1 16 129 113 0.441369 0.441406 0.100164 0.129642 2023-11-02T09:03:55.552 INFO:tasks.workunit.client.0.smithi116.stdout: 2 16 288 272 0.531192 0.621094 0.117479 0.11509 2023-11-02T09:03:56.552 INFO:tasks.workunit.client.0.smithi116.stdout: 3 16 448 432 0.562432 0.625 0.0756337 0.110647 2023-11-02T09:03:57.553 INFO:tasks.workunit.client.0.smithi116.stdout: 4 16 608 592 0.57805 0.625 0.128551 0.106954 2023-11-02T09:03:58.553 INFO:tasks.workunit.client.0.smithi116.stdout: 5 16 752 736 0.574931 0.5625 0.0839083 0.108617 2023-11-02T09:03:59.553 INFO:tasks.workunit.client.0.smithi116.stdout: 6 16 897 881 0.573502 0.566406 0.117345 0.107436 2023-11-02T09:04:00.553 INFO:tasks.workunit.client.0.smithi116.stdout: 7 16 1008 992 0.553506 0.433594 0.0926049 0.112721 2023-11-02T09:04:01.553 INFO:tasks.workunit.client.0.smithi116.stdout: 8 16 1184 1168 0.570242 0.6875 0.108788 0.109265 2023-11-02T09:04:02.553 INFO:tasks.workunit.client.0.smithi116.stdout: 9 16 1313 1297 0.562864 0.503906 0.0838925 0.109932 2023-11-02T09:04:03.553 INFO:tasks.workunit.client.0.smithi116.stdout: 10 16 1489 1473 0.575317 0.6875 0.10075 0.108082 2023-11-02T09:04:04.553 INFO:tasks.workunit.client.0.smithi116.stdout: 11 16 1633 1617 0.574143 0.5625 0.118332 0.10824 2023-11-02T09:04:05.554 INFO:tasks.workunit.client.0.smithi116.stdout: 12 16 1745 1729 0.562751 0.4375 0.117379 0.110204 2023-11-02T09:04:06.554 INFO:tasks.workunit.client.0.smithi116.stdout: 13 16 1889 1873 0.562727 0.5625 0.100299 0.110311 2023-11-02T09:04:07.554 INFO:tasks.workunit.client.0.smithi116.stdout: 14 16 2049 2033 0.56717 0.625 0.108805 0.109801 2023-11-02T09:04:08.554 INFO:tasks.workunit.client.0.smithi116.stdout: 15 16 2193 2177 0.566856 0.5625 0.091882 0.109862 2023-11-02T09:04:09.554 INFO:tasks.workunit.client.0.smithi116.stdout: 16 16 2352 2336 0.570242 0.621094 0.100675 0.109451 2023-11-02T09:04:10.554 INFO:tasks.workunit.client.0.smithi116.stdout: 17 16 2481 2465 0.566337 0.503906 0.10901 0.109966 2023-11-02T09:04:11.554 INFO:tasks.workunit.client.0.smithi116.stdout: 18 16 2641 2625 0.56959 0.625 0.0836 0.109428 2023-11-02T09:04:12.554 INFO:tasks.workunit.client.0.smithi116.stdout: 19 16 2769 2753 0.565925 0.5 0.12575 0.109802 2023-11-02T09:04:13.554 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T09:04:13.552874+0000 min lat: 0.0508074 max lat: 0.372549 avg lat: 0.11002 2023-11-02T09:04:13.555 INFO:tasks.workunit.client.0.smithi116.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-11-02T09:04:13.555 INFO:tasks.workunit.client.0.smithi116.stdout: 20 16 2913 2897 0.565751 0.5625 0.14195 0.11002 2023-11-02T09:04:14.554 INFO:tasks.workunit.client.0.smithi116.stdout: 21 16 3072 3056 0.568383 0.621094 0.117126 0.109923 2023-11-02T09:04:15.555 INFO:tasks.workunit.client.0.smithi116.stdout: 22 16 3232 3216 0.570954 0.625 0.0839688 0.109202 2023-11-02T09:04:16.555 INFO:tasks.workunit.client.0.smithi116.stdout: 23 16 3376 3360 0.570584 0.5625 0.100487 0.109505 2023-11-02T09:04:17.559 INFO:tasks.workunit.client.0.smithi116.stdout: 24 16 3521 3505 0.570408 0.566406 0.108807 0.109247 2023-11-02T09:04:18.555 INFO:tasks.workunit.client.0.smithi116.stdout: 25 16 3665 3649 0.57009 0.5625 0.0920829 0.109377 2023-11-02T09:04:19.555 INFO:tasks.workunit.client.0.smithi116.stdout: 26 16 3840 3824 0.574452 0.683594 0.0757765 0.108694 2023-11-02T09:04:20.555 INFO:tasks.workunit.client.0.smithi116.stdout: 27 16 3969 3953 0.571836 0.503906 0.0921408 0.109066 2023-11-02T09:04:21.555 INFO:tasks.workunit.client.0.smithi116.stdout: 28 16 4144 4128 0.575824 0.683594 0.084117 0.108511 2023-11-02T09:04:22.556 INFO:tasks.workunit.client.0.smithi116.stdout: 29 16 4273 4257 0.573342 0.503906 0.133961 0.108773 2023-11-02T09:04:23.556 INFO:tasks.workunit.client.0.smithi116.stdout: 30 1 4433 4432 0.577015 0.683594 0.0919581 0.10829 2023-11-02T09:04:24.556 INFO:tasks.workunit.client.0.smithi116.stdout:Total time run: 30.0257 2023-11-02T09:04:24.556 INFO:tasks.workunit.client.0.smithi116.stdout:Total writes made: 4433 2023-11-02T09:04:24.556 INFO:tasks.workunit.client.0.smithi116.stdout:Write size: 4096 2023-11-02T09:04:24.557 INFO:tasks.workunit.client.0.smithi116.stdout:Object size: 4096 2023-11-02T09:04:24.557 INFO:tasks.workunit.client.0.smithi116.stdout:Bandwidth (MB/sec): 0.57672 2023-11-02T09:04:24.557 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev Bandwidth: 0.0750326 2023-11-02T09:04:24.557 INFO:tasks.workunit.client.0.smithi116.stdout:Max bandwidth (MB/sec): 0.6875 2023-11-02T09:04:24.557 INFO:tasks.workunit.client.0.smithi116.stdout:Min bandwidth (MB/sec): 0.433594 2023-11-02T09:04:24.557 INFO:tasks.workunit.client.0.smithi116.stdout:Average IOPS: 147 2023-11-02T09:04:24.558 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev IOPS: 19.2084 2023-11-02T09:04:24.558 INFO:tasks.workunit.client.0.smithi116.stdout:Max IOPS: 176 2023-11-02T09:04:24.558 INFO:tasks.workunit.client.0.smithi116.stdout:Min IOPS: 111 2023-11-02T09:04:24.558 INFO:tasks.workunit.client.0.smithi116.stdout:Average Latency(s): 0.108285 2023-11-02T09:04:24.559 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev Latency(s): 0.0328182 2023-11-02T09:04:24.559 INFO:tasks.workunit.client.0.smithi116.stdout:Max latency(s): 0.372549 2023-11-02T09:04:24.559 INFO:tasks.workunit.client.0.smithi116.stdout:Min latency(s): 0.0508074 2023-11-02T09:04:24.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:433: TEST_bluestore_expand: sleep 5 2023-11-02T09:04:29.617 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2023-11-02T09:04:29.617 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2023-11-02T09:04:29.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: total_space_before=4294967296 2023-11-02T09:04:29.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2023-11-02T09:04:29.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2023-11-02T09:04:29.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep free 2023-11-02T09:04:29.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d: -f 2 2023-11-02T09:04:29.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d, -f 1 2023-11-02T09:04:29.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut '-d ' -f 2 2023-11-02T09:04:29.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: free_space_before=4276707328 2023-11-02T09:04:29.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 207217 2023-11-02T09:04:29.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: sleep 1 2023-11-02T09:04:30.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 207217 2023-11-02T09:04:30.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 439: kill: (207217) - No such process 2023-11-02T09:04:30.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:440: TEST_bluestore_expand: ceph osd down 0 2023-11-02T09:04:31.299 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 is already down. 2023-11-02T09:04:31.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 allocmap 2023-11-02T09:04:31.337 INFO:tasks.workunit.client.0.smithi116.stdout:allocmap bluestore.allocmap 2023-11-02T09:04:34.101 INFO:tasks.workunit.client.0.smithi116.stdout:allocmap success 2023-11-02T09:04:34.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:446: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 fsck 2023-11-02T09:04:36.901 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T09:04:36.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:448: TEST_bluestore_expand: requested_space=4294967296 2023-11-02T09:04:36.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:449: TEST_bluestore_expand: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2023-11-02T09:04:36.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:450: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2023-11-02T09:04:36.941 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T09:04:38.988 INFO:tasks.workunit.client.0.smithi116.stdout:1 : device size 0x3fffe000 : using 0x2200000(34 MiB) 2023-11-02T09:04:38.988 INFO:tasks.workunit.client.0.smithi116.stdout:2 : device size 0x100000000 : using 0x116a000(17 MiB) 2023-11-02T09:04:38.989 INFO:tasks.workunit.client.0.smithi116.stdout:Expanding DB/WAL... 2023-11-02T09:04:38.989 INFO:tasks.workunit.client.0.smithi116.stdout:1 : expanding to 0x1073741824 2023-11-02T09:04:39.017 INFO:tasks.workunit.client.0.smithi116.stdout:1 : size label updated to 1073741824 2023-11-02T09:04:39.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:453: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 fsck 2023-11-02T09:04:42.510 INFO:tasks.workunit.client.0.smithi116.stdout:fsck success 2023-11-02T09:04:42.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:456: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 qfsck 2023-11-02T09:04:42.546 INFO:tasks.workunit.client.0.smithi116.stdout:qfsck bluestore.quick-fsck 2023-11-02T09:04:45.309 INFO:tasks.workunit.client.0.smithi116.stdout:qfsck success 2023-11-02T09:04:45.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:458: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2023-11-02T09:04:45.343 INFO:tasks.workunit.client.0.smithi116.stdout:inferring bluefs devices from bluestore path 2023-11-02T09:04:48.121 INFO:tasks.workunit.client.0.smithi116.stdout:1 : device size 0x3fffe000 : using 0x2200000(34 MiB) 2023-11-02T09:04:48.121 INFO:tasks.workunit.client.0.smithi116.stdout:2 : device size 0x100000000 : using 0x116a000(17 MiB) 2023-11-02T09:04:48.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:460: TEST_bluestore_expand: activate_osd td/osd-bluefs-volume-ops 0 2023-11-02T09:04:48.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:04:48.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T09:04:48.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-11-02T09:04:48.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T09:04:48.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2023-11-02T09:04:48.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=3aaadb38-61d9-40f4-8cb1-362d1e219519 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=c6749193-4650-4de3-9cb2-c42bf836b301 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2023-11-02T09:04:48.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:04:48.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:04:48.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:04:48.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2023-11-02T09:04:48.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2023-11-02T09:04:48.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T09:04:48.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T09:04:48.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2023-11-02T09:04:48.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T09:04:48.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:04:48.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:04:48.131 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:04:48.131 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:04:48.131 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:04:48.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T09:04:48.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' 2023-11-02T09:04:48.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:04:48.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2023-11-02T09:04:48.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2023-11-02T09:04:48.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:04:48.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:04:48.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:04:48.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:04:48.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T09:04:48.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T09:04:48.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2023-11-02T09:04:48.134 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T09:04:48.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-11-02T09:04:48.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=cc15c2b6-022e-44b4-9dbe-56b3424aaa74 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=3aaadb38-61d9-40f4-8cb1-362d1e219519 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=c6749193-4650-4de3-9cb2-c42bf836b301 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.153754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:04:48.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2023-11-02T09:04:48.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-11-02T09:04:48.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T09:04:48.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T09:04:48.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T09:04:48.168 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:04:48.165+0000 7f83abe16500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:04:48.171 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:04:48.168+0000 7f83abe16500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:04:48.172 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:04:48.169+0000 7f83abe16500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:04:48.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-11-02T09:04:48.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:04:48.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T09:04:48.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:04:48.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:04:48.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:04:48.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:04:48.525 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:04:48.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:04:48.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:04:48.940 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:04:48.937+0000 7f83abe16500 -1 Falling back to public interface 2023-11-02T09:04:48.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:04:49.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:04:49.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:04:49.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:04:49.946 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:04:49.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:04:49.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:04:50.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:04:51.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:04:51.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:04:51.396 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:04:51.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:04:51.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:04:51.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:04:51.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:04:52.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:04:52.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:04:52.775 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:04:52.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:04:52.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:04:52.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:04:53.069 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:04:53.066+0000 7f83abe16500 -1 osd.0 10 log_to_monitors true 2023-11-02T09:04:53.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:04:54.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:04:54.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:04:54.159 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:04:54.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:04:54.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:04:54.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:04:55.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:04:56.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:04:56.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:04:56.056 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:04:56.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:04:56.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:04:56.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:04:56.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:04:57.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:04:57.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:04:57.535 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:04:57.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:04:57.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:04:57.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:04:57.968 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 13 up_thru 13 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/210569,v1:127.0.0.1:6803/210569] [v2:127.0.0.1:6804/210569,v1:127.0.0.1:6805/210569] exists,up f96ec9f8-f335-4060-826e-02fd5f4b07ba 2023-11-02T09:04:57.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:04:57.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:04:57.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:04:57.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:461: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2023-11-02T09:04:57.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:461: TEST_bluestore_expand: osd_pid0=210569 2023-11-02T09:04:57.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:463: TEST_bluestore_expand: wait_for_clean 2023-11-02T09:04:57.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T09:04:57.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T09:04:57.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T09:04:57.972 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-11-02T09:04:57.972 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T09:04:57.972 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T09:04:57.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T09:04:57.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T09:04:57.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T09:04:58.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-11-02T09:04:58.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T09:04:58.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T09:04:58.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T09:04:58.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T09:04:58.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T09:04:58.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2023-11-02T09:04:58.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T09:04:58.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:04:58.435 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T09:04:58.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=55834574852 2023-11-02T09:04:58.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 55834574852 2023-11-02T09:04:58.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-55834574852' 2023-11-02T09:04:58.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:04:58.557 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-55834574852 2023-11-02T09:04:58.558 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:04:58.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T09:04:58.559 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-55834574852 2023-11-02T09:04:58.559 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:04:58.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=55834574852 2023-11-02T09:04:58.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2023-11-02T09:04:58.560 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 55834574852 2023-11-02T09:04:58.561 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:04:58.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 1 -lt 55834574852 2023-11-02T09:04:58.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T09:04:59.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T09:04:59.972 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:05:00.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 55834574852 -lt 55834574852 2023-11-02T09:05:00.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T09:05:00.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:05:00.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:05:00.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2023-11-02T09:05:00.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:05:00.898 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:05:00.898 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:05:00.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:05:00.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:05:00.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:05:00.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:05:01.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2023-11-02T09:05:01.306 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:05:01.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:05:01.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:05:01.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2023-11-02T09:05:01.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T09:05:01.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T09:05:01.825 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2023-11-02T09:05:01.825 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2023-11-02T09:05:01.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: total_space_after=4294967296 2023-11-02T09:05:01.971 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2023-11-02T09:05:01.972 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2023-11-02T09:05:01.972 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: grep free 2023-11-02T09:05:01.972 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut -d: -f 2 2023-11-02T09:05:01.972 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut -d, -f 1 2023-11-02T09:05:01.972 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut '-d ' -f 2 2023-11-02T09:05:02.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: free_space_after=4276674560 2023-11-02T09:05:02.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:468: TEST_bluestore_expand: '[4294967296' '!=' '4294967296]' 2023-11-02T09:05:02.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 468: [4294967296: command not found 2023-11-02T09:05:02.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:474: TEST_bluestore_expand: total_space_added=0 2023-11-02T09:05:02.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:475: TEST_bluestore_expand: free_space_added=-32768 2023-11-02T09:05:02.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:477: TEST_bluestore_expand: let 'new_used_space=(0 - -32768)' 2023-11-02T09:05:02.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:478: TEST_bluestore_expand: echo 32768 2023-11-02T09:05:02.141 INFO:tasks.workunit.client.0.smithi116.stdout:32768 2023-11-02T09:05:02.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:480: TEST_bluestore_expand: '[' 32768 -gt 131072 ']' 2023-11-02T09:05:02.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: kill 210569 2023-11-02T09:05:02.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: sleep 1 2023-11-02T09:05:03.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: kill 210569 2023-11-02T09:05:03.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 487: kill: (210569) - No such process 2023-11-02T09:05:03.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:488: TEST_bluestore_expand: ceph osd down 0 2023-11-02T09:05:03.949 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 is already down. 2023-11-02T09:05:03.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:490: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 qfsck 2023-11-02T09:05:03.991 INFO:tasks.workunit.client.0.smithi116.stdout:qfsck bluestore.quick-fsck 2023-11-02T09:05:06.758 INFO:tasks.workunit.client.0.smithi116.stdout:qfsck success 2023-11-02T09:05:06.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2023-11-02T09:05:06.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:05:06.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:05:06.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2023-11-02T09:05:06.765 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:05:06.765 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:05:06.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:05:06.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:05:06.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:05:06.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:05:06.872 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:05:06.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:05:06.874 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:05:06.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:05:06.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:05:06.875 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:05:06.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:05:06.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:05:06.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:05:06.877 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:05:06.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:05:06.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:05:06.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2023-11-02T09:05:06.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:05:06.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:05:06.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:05:06.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153754 2023-11-02T09:05:06.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:05:06.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:05:06.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2023-11-02T09:05:06.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-bluefs-volume-ops 0 2023-11-02T09:05:06.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2023-11-02T09:05:06.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-11-02T09:05:06.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2023-11-02T09:05:06.958 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:05:06.958 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:05:06.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:05:06.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:05:06.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:05:06.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:05:06.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:05:06.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:05:06.960 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:05:06.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:05:06.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:05:06.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:05:06.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:05:06.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:05:06.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:05:06.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:05:06.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:05:06.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-11-02T09:05:06.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2023-11-02T09:05:06.966 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:05:06.966 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:05:06.966 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153754 2023-11-02T09:05:06.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153754 2023-11-02T09:05:06.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:05:06.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:05:06.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2023-11-02T09:05:06.969 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-02T09:05:06.969 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-02T09:05:07.001 INFO:tasks.workunit:Running workunit osd/osd-config.sh... 2023-11-02T09:05:07.002 DEBUG:teuthology.orchestra.run.smithi116:workunit test osd/osd-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh 2023-11-02T09:05:07.064 INFO:tasks.workunit.client.0.smithi116.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-11-02T09:05:07.070 INFO:tasks.workunit.client.0.smithi116.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-11-02T09:05:07.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T09:05:07.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T09:05:07.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2023-11-02T09:05:07.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2023-11-02T09:05:07.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2023-11-02T09:05:07.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2023-11-02T09:05:07.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2023-11-02T09:05:07.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2023-11-02T09:05:07.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-config 2023-11-02T09:05:07.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:22: run: local dir=td/osd-config 2023-11-02T09:05:07.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:23: run: shift 2023-11-02T09:05:07.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: export CEPH_MON=127.0.0.1:7100 2023-11-02T09:05:07.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: CEPH_MON=127.0.0.1:7100 2023-11-02T09:05:07.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:26: run: export CEPH_ARGS 2023-11-02T09:05:07.074 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: uuidgen 2023-11-02T09:05:07.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: CEPH_ARGS+='--fsid=dea85ac7-3b7d-4fb0-a3ed-75f656a3a986 --auth-supported=none ' 2023-11-02T09:05:07.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2023-11-02T09:05:07.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: set 2023-11-02T09:05:07.076 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-11-02T09:05:07.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: local 'funcs=TEST_config_init 2023-11-02T09:05:07.079 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_config_track 2023-11-02T09:05:07.079 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_default_adjustment' 2023-11-02T09:05:07.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2023-11-02T09:05:07.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2023-11-02T09:05:07.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2023-11-02T09:05:07.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2023-11-02T09:05:07.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2023-11-02T09:05:07.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:05:07.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2023-11-02T09:05:07.081 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:05:07.081 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:05:07.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:05:07.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:05:07.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:05:07.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:05:07.082 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:05:07.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:05:07.082 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:05:07.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:05:07.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:05:07.083 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:05:07.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:05:07.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:05:07.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:05:07.084 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:05:07.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:05:07.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:05:07.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2023-11-02T09:05:07.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:05:07.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:05:07.088 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:05:07.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.212321 2023-11-02T09:05:07.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:05:07.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:05:07.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2023-11-02T09:05:07.089 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T09:05:07.089 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:05:07.089 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:05:07.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.212321 2023-11-02T09:05:07.089 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T09:05:07.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-11-02T09:05:07.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-11-02T09:05:07.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T09:05:07.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2023-11-02T09:05:07.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_init td/osd-config 2023-11-02T09:05:07.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:39: TEST_config_init: local dir=td/osd-config 2023-11-02T09:05:07.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:41: TEST_config_init: run_mon td/osd-config a 2023-11-02T09:05:07.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2023-11-02T09:05:07.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T09:05:07.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T09:05:07.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T09:05:07.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2023-11-02T09:05:07.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2023-11-02T09:05:07.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T09:05:07.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:05:07.460 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:05:07.460 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:05:07.460 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:05:07.460 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:05:07.460 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.212321/$cluster-$name.asok' 2023-11-02T09:05:07.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.212321/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T09:05:07.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T09:05:07.645 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T09:05:07.645 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:05:07.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:05:07.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T09:05:07.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T09:05:07.646 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:05:07.646 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:05:07.647 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:05:07.647 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:05:07.647 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:05:07.647 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:05:07.647 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.212321/ceph-mon.a.asok 2023-11-02T09:05:07.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:05:07.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.212321/ceph-mon.a.asok config get fsid 2023-11-02T09:05:07.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T09:05:07.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:05:07.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:05:07.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T09:05:07.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T09:05:07.727 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:05:07.727 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:05:07.727 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:05:07.728 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:05:07.728 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:05:07.728 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:05:07.728 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.212321/ceph-mon.a.asok 2023-11-02T09:05:07.728 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:05:07.728 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.212321/ceph-mon.a.asok config get mon_host 2023-11-02T09:05:07.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:42: TEST_config_init: run_mgr td/osd-config x 2023-11-02T09:05:07.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2023-11-02T09:05:07.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T09:05:07.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T09:05:07.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T09:05:07.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2023-11-02T09:05:07.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T09:05:08.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T09:05:08.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:05:08.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:05:08.650 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:05:08.650 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:05:08.650 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:05:08.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.212321/$cluster-$name.asok' 2023-11-02T09:05:08.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:05:08.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.212321/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:05:08.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:43: TEST_config_init: local stale=1000 2023-11-02T09:05:08.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:44: TEST_config_init: local cache=500 2023-11-02T09:05:08.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:45: TEST_config_init: run_osd td/osd-config 0 --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2023-11-02T09:05:08.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2023-11-02T09:05:08.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:05:08.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T09:05:08.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:05:08.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2023-11-02T09:05:08.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dea85ac7-3b7d-4fb0-a3ed-75f656a3a986 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2023-11-02T09:05:08.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:05:08.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:05:08.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:05:08.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2023-11-02T09:05:08.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2023-11-02T09:05:08.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:05:08.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:05:08.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2023-11-02T09:05:08.682 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:05:08.682 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:05:08.682 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:05:08.682 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:05:08.682 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:05:08.683 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:05:08.683 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.212321/$cluster-$name.asok' 2023-11-02T09:05:08.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.212321/$cluster-$name.asok' 2023-11-02T09:05:08.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:05:08.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:05:08.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:05:08.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2023-11-02T09:05:08.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2023-11-02T09:05:08.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:05:08.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:05:08.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:05:08.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:05:08.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:05:08.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000' 2023-11-02T09:05:08.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-config/0 2023-11-02T09:05:08.685 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:05:08.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e549c40e-1e82-4ebd-b8b2-aadf5a7948cc 2023-11-02T09:05:08.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e549c40e-1e82-4ebd-b8b2-aadf5a7948cc' 2023-11-02T09:05:08.686 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 e549c40e-1e82-4ebd-b8b2-aadf5a7948cc 2023-11-02T09:05:08.687 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:05:08.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBEZkNle4I2KRAACr2ORulsLH9gqO6rhZCFzg== 2023-11-02T09:05:08.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBEZkNle4I2KRAACr2ORulsLH9gqO6rhZCFzg=="}' 2023-11-02T09:05:08.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e549c40e-1e82-4ebd-b8b2-aadf5a7948cc -i td/osd-config/0/new.json 2023-11-02T09:05:09.044 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:05:09.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-config/0/new.json 2023-11-02T09:05:09.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=dea85ac7-3b7d-4fb0-a3ed-75f656a3a986 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.212321/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 --mkfs --key AQBEZkNle4I2KRAACr2ORulsLH9gqO6rhZCFzg== --osd-uuid e549c40e-1e82-4ebd-b8b2-aadf5a7948cc 2023-11-02T09:05:09.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:09.078+0000 7f7ed6e0f500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:05:09.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:09.080+0000 7f7ed6e0f500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:05:09.084 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:09.082+0000 7f7ed6e0f500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:05:09.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:09.082+0000 7f7ed6e0f500 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2023-11-02T09:05:09.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:09.082+0000 7f7ed6e0f500 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2023-11-02T09:05:09.085 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:09.082+0000 7f7ed6e0f500 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2023-11-02T09:05:09.159 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:09.157+0000 7f7ed6e0f500 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2023-11-02T09:05:14.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-config/0/keyring 2023-11-02T09:05:14.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:05:14.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T09:05:14.490 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T09:05:14.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:05:15.008 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T09:05:15.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T09:05:15.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=dea85ac7-3b7d-4fb0-a3ed-75f656a3a986 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.212321/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2023-11-02T09:05:15.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:05:15.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:05:15.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:05:15.033 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:15.030+0000 7f82d1d12500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:05:15.035 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:15.032+0000 7f82d1d12500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:05:15.036 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:15.033+0000 7f82d1d12500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:05:15.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T09:05:15.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:05:15.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T09:05:15.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:05:15.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:05:15.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:15.389 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:05:15.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:05:15.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:15.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:15.555 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:15.553+0000 7f82d1d12500 -1 Falling back to public interface 2023-11-02T09:05:15.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:16.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:16.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:16.792 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:05:16.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:05:16.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:16.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:17.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:17.861 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:17.858+0000 7f82d1d12500 -1 osd.0 0 log_to_monitors true 2023-11-02T09:05:18.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:18.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:18.193 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:05:18.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:05:18.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:18.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:18.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:19.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:19.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:19.614 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:05:19.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:05:19.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:19.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:20.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:21.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:21.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:21.020 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:05:21.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:05:21.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:21.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:21.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:22.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:22.620 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:05:22.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:22.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:05:22.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:22.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:22.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:24.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:24.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:24.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:05:24.001 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:05:24.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:24.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:24.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:25.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:25.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:25.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:05:25.429 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:05:25.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:25.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:25.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:26.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:26.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:26.926 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:05:26.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:05:26.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:26.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:27.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:28.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:28.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:28.356 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:05:28.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:05:28.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:28.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:28.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:29.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:29.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:29.738 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:05:29.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:05:29.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:29.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:30.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:31.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:31.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:31.474 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:05:31.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:05:31.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:31.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:32.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:33.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:33.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:33.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:05:33.022 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:05:33.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:33.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:34.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:35.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:35.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:35.035 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:05:35.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:05:35.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:35.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:35.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:36.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:36.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:36.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:05:36.928 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:05:36.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:36.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:37.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:38.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:38.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:38.967 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:05:38.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:05:38.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:38.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:40.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:41.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:41.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:41.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:05:41.050 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:05:41.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:41.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:41.518 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/213405,v1:127.0.0.1:6803/213405] [v2:127.0.0.1:6804/213405,v1:127.0.0.1:6805/213405] exists,up e549c40e-1e82-4ebd-b8b2-aadf5a7948cc 2023-11-02T09:05:41.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:05:41.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:05:41.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:05:41.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: get_asok_path osd.0 2023-11-02T09:05:41.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-11-02T09:05:41.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-11-02T09:05:41.521 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:05:41.521 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:05:41.521 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:05:41.521 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.212321/ceph-osd.0.asok 2023-11-02T09:05:41.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: CEPH_ARGS= 2023-11-02T09:05:41.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: ceph --admin-daemon /tmp/ceph-asok.212321/ceph-osd.0.asok log flush 2023-11-02T09:05:41.598 INFO:tasks.workunit.client.0.smithi116.stdout:{} 2023-11-02T09:05:41.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:50: TEST_config_init: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2023-11-02T09:05:41.606 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T09:05:18.223+0000 7f82d1d12500 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2023-11-02T09:05:41.606 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T09:05:20.858+0000 7f82c1c9b640 10 log_client will send 2023-11-02T09:05:18.224242+0000 osd.0 (osd.0) 1 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2023-11-02T09:05:41.606 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T09:05:21.170+0000 7f82c2c9d640 10 log_client logged 2023-11-02T09:05:18.224242+0000 osd.0 (osd.0) 1 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2023-11-02T09:05:41.606 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T09:05:40.132+0000 7f82cdcb3640 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2023-11-02T09:05:41.607 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T09:05:40.861+0000 7f82c1c9b640 10 log_client will send 2023-11-02T09:05:40.133807+0000 osd.0 (osd.0) 4 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2023-11-02T09:05:41.607 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T09:05:41.044+0000 7f82c2c9d640 10 log_client logged 2023-11-02T09:05:40.133807+0000 osd.0 (osd.0) 4 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2023-11-02T09:05:41.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2023-11-02T09:05:41.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2023-11-02T09:05:41.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:05:41.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2023-11-02T09:05:41.610 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:05:41.610 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:05:41.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:05:41.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:05:41.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:05:41.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:05:41.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:05:41.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:05:41.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:05:41.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:05:41.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:05:41.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:05:41.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:05:41.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:05:41.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:05:41.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:05:41.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:05:41.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:05:41.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2023-11-02T09:05:41.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:05:41.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:05:41.723 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:05:41.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.212321 2023-11-02T09:05:41.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:05:41.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:05:41.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2023-11-02T09:05:41.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2023-11-02T09:05:41.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2023-11-02T09:05:41.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2023-11-02T09:05:41.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2023-11-02T09:05:41.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:05:41.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2023-11-02T09:05:41.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:05:41.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:05:41.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:05:41.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:05:41.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:05:41.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:05:41.729 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:05:41.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:05:41.729 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:05:41.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:05:41.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:05:41.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:05:41.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:05:41.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:05:41.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:05:41.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:05:41.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:05:41.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:05:41.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2023-11-02T09:05:41.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:05:41.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:05:41.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:05:41.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.212321 2023-11-02T09:05:41.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:05:41.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:05:41.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2023-11-02T09:05:41.734 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T09:05:41.734 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:05:41.735 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:05:41.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.212321 2023-11-02T09:05:41.736 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T09:05:41.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-11-02T09:05:41.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T09:05:41.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2023-11-02T09:05:41.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_track td/osd-config 2023-11-02T09:05:41.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:54: TEST_config_track: local dir=td/osd-config 2023-11-02T09:05:41.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:56: TEST_config_track: run_mon td/osd-config a 2023-11-02T09:05:41.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2023-11-02T09:05:41.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T09:05:41.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T09:05:41.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T09:05:41.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2023-11-02T09:05:41.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2023-11-02T09:05:42.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T09:05:42.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:05:42.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:05:42.139 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:05:42.139 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:05:42.139 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:05:42.139 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.212321/$cluster-$name.asok' 2023-11-02T09:05:42.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.212321/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T09:05:42.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T09:05:42.324 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T09:05:42.324 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:05:42.324 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:05:42.325 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T09:05:42.325 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T09:05:42.325 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:05:42.325 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:05:42.325 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:05:42.326 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:05:42.326 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:05:42.326 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:05:42.326 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.212321/ceph-mon.a.asok 2023-11-02T09:05:42.326 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:05:42.327 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.212321/ceph-mon.a.asok config get fsid 2023-11-02T09:05:42.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T09:05:42.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:05:42.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:05:42.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T09:05:42.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T09:05:42.408 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:05:42.408 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:05:42.408 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:05:42.409 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:05:42.409 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:05:42.409 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:05:42.409 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.212321/ceph-mon.a.asok 2023-11-02T09:05:42.409 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:05:42.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.212321/ceph-mon.a.asok config get mon_host 2023-11-02T09:05:42.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:57: TEST_config_track: run_mgr td/osd-config x 2023-11-02T09:05:42.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2023-11-02T09:05:42.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T09:05:42.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T09:05:42.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T09:05:42.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2023-11-02T09:05:42.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T09:05:43.327 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T09:05:43.327 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:05:43.328 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:05:43.328 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:05:43.328 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:05:43.328 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:05:43.328 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.212321/$cluster-$name.asok' 2023-11-02T09:05:43.329 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:05:43.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.212321/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:05:43.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:58: TEST_config_track: run_osd td/osd-config 0 2023-11-02T09:05:43.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2023-11-02T09:05:43.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:05:43.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T09:05:43.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:05:43.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2023-11-02T09:05:43.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dea85ac7-3b7d-4fb0-a3ed-75f656a3a986 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2023-11-02T09:05:43.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:05:43.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:05:43.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:05:43.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2023-11-02T09:05:43.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2023-11-02T09:05:43.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:05:43.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:05:43.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2023-11-02T09:05:43.359 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:05:43.359 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:05:43.359 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:05:43.359 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:05:43.360 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:05:43.360 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:05:43.360 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.212321/$cluster-$name.asok' 2023-11-02T09:05:43.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.212321/$cluster-$name.asok' 2023-11-02T09:05:43.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:05:43.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:05:43.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:05:43.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2023-11-02T09:05:43.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2023-11-02T09:05:43.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:05:43.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:05:43.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:05:43.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:05:43.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:05:43.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:05:43.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-config/0 2023-11-02T09:05:43.363 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:05:43.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2a4c707d-cba4-46a5-9768-cae7dcebc62d 2023-11-02T09:05:43.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2a4c707d-cba4-46a5-9768-cae7dcebc62d' 2023-11-02T09:05:43.363 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 2a4c707d-cba4-46a5-9768-cae7dcebc62d 2023-11-02T09:05:43.364 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:05:43.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBnZkNlNv0AFhAAc1jRI6BPe3Uf4ZBDZ4iATA== 2023-11-02T09:05:43.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBnZkNlNv0AFhAAc1jRI6BPe3Uf4ZBDZ4iATA=="}' 2023-11-02T09:05:43.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2a4c707d-cba4-46a5-9768-cae7dcebc62d -i td/osd-config/0/new.json 2023-11-02T09:05:43.663 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:05:43.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-config/0/new.json 2023-11-02T09:05:43.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=dea85ac7-3b7d-4fb0-a3ed-75f656a3a986 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.212321/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBnZkNlNv0AFhAAc1jRI6BPe3Uf4ZBDZ4iATA== --osd-uuid 2a4c707d-cba4-46a5-9768-cae7dcebc62d 2023-11-02T09:05:43.699 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:43.696+0000 7f2fcca17500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:05:43.715 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:43.712+0000 7f2fcca17500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:05:43.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:43.713+0000 7f2fcca17500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:05:43.716 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:43.714+0000 7f2fcca17500 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2023-11-02T09:05:43.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:43.714+0000 7f2fcca17500 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2023-11-02T09:05:43.717 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:43.714+0000 7f2fcca17500 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2023-11-02T09:05:43.755 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:43.752+0000 7f2fcca17500 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2023-11-02T09:05:49.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-config/0/keyring 2023-11-02T09:05:49.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:05:49.125 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T09:05:49.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T09:05:49.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:05:49.649 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T09:05:49.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T09:05:49.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=dea85ac7-3b7d-4fb0-a3ed-75f656a3a986 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.212321/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:05:49.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:05:49.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:05:49.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:05:49.672 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:49.670+0000 7f1984616500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:05:49.686 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:49.682+0000 7f1984616500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:05:49.686 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:49.684+0000 7f1984616500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:05:50.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T09:05:50.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:05:50.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T09:05:50.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:05:50.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:05:50.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:50.084 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:05:50.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:05:50.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:50.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:50.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:50.214+0000 7f1984616500 -1 Falling back to public interface 2023-11-02T09:05:50.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:51.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:51.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:51.484 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:05:51.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:05:51.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:51.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:51.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:52.549 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:05:52.546+0000 7f1984616500 -1 osd.0 0 log_to_monitors true 2023-11-02T09:05:52.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:52.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:52.903 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:05:52.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:05:52.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:52.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:53.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:54.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:54.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:54.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:05:54.299 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:05:54.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:54.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:54.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:55.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:55.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:55.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:05:55.678 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:05:55.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:55.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:56.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:57.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:57.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:57.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:05:57.075 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:05:57.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:57.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:57.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:05:58.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:05:58.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:05:58.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:05:58.606 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:05:58.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:05:58.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:05:59.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:00.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:00.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:00.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:06:00.030 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:06:00.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:00.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:00.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:01.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:01.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:01.425 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:06:01.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:06:01.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:01.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:01.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:02.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:02.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:02.814 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:06:02.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:06:02.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:02.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:03.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:04.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:04.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:04.232 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:06:04.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:06:04.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:04.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:04.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:05.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:05.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:05.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:06:05.616 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:06:05.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:05.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:06.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:07.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:07.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:07.172 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:06:07.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:06:07.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:07.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:08.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:09.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:09.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:09.156 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:06:09.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:06:09.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:09.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:10.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:11.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:11.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:11.174 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:06:11.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:06:11.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:11.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:12.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:13.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:13.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:13.111 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:06:13.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:06:13.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:13.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:13.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:14.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:14.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:14.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:06:14.911 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:06:14.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:14.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:15.307 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/215389,v1:127.0.0.1:6803/215389] [v2:127.0.0.1:6804/215389,v1:127.0.0.1:6805/215389] exists,up 2a4c707d-cba4-46a5-9768-cae7dcebc62d 2023-11-02T09:06:15.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:06:15.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:06:15.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:06:15.308 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: CEPH_ARGS= 2023-11-02T09:06:15.308 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: ceph-conf --show-config-value osd_map_cache_size 2023-11-02T09:06:15.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: local osd_map_cache_size=50 2023-11-02T09:06:15.321 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: CEPH_ARGS= 2023-11-02T09:06:15.321 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: ceph-conf --show-config-value osd_pg_epoch_persisted_max_stale 2023-11-02T09:06:15.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: local osd_pg_epoch_persisted_max_stale=40 2023-11-02T09:06:15.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:68: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2023-11-02T09:06:15.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:69: TEST_config_track: local stale=100 2023-11-02T09:06:15.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:70: TEST_config_track: ceph tell osd.0 injectargs '--osd-pg-epoch-persisted-max-stale 100' 2023-11-02T09:06:15.439 INFO:tasks.workunit.client.0.smithi116.stdout:{} 2023-11-02T09:06:15.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: get_asok_path osd.0 2023-11-02T09:06:15.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-11-02T09:06:15.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-11-02T09:06:15.449 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:06:15.450 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:06:15.450 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:06:15.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.212321/ceph-osd.0.asok 2023-11-02T09:06:15.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: CEPH_ARGS= 2023-11-02T09:06:15.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.212321/ceph-osd.0.asok log flush 2023-11-02T09:06:15.529 INFO:tasks.workunit.client.0.smithi116.stdout:{} 2023-11-02T09:06:15.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:72: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2023-11-02T09:06:15.537 INFO:tasks.workunit.client.0.smithi116.stdout:2023-11-02T09:06:15.425+0000 7f19827d9640 0 log_channel(cluster) log [WRN] : osd_map_cache_size (50) is not > osd_pg_epoch_persisted_max_stale (100) 2023-11-02T09:06:15.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:73: TEST_config_track: rm td/osd-config/osd.0.log 2023-11-02T09:06:15.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: get_asok_path osd.0 2023-11-02T09:06:15.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-11-02T09:06:15.540 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-11-02T09:06:15.540 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:06:15.540 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:06:15.540 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:06:15.540 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.212321/ceph-osd.0.asok 2023-11-02T09:06:15.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: CEPH_ARGS= 2023-11-02T09:06:15.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.212321/ceph-osd.0.asok log reopen 2023-11-02T09:06:15.618 INFO:tasks.workunit.client.0.smithi116.stdout:{} 2023-11-02T09:06:15.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2023-11-02T09:06:15.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2023-11-02T09:06:15.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:06:15.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2023-11-02T09:06:15.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:06:15.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:06:15.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:06:15.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:06:15.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:06:15.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:06:15.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:06:15.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:06:15.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:06:15.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:06:15.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:06:15.734 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:06:15.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:06:15.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:06:15.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:06:15.735 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:06:15.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:06:15.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:06:15.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2023-11-02T09:06:15.741 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:06:15.741 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:06:15.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:06:15.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.212321 2023-11-02T09:06:15.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:06:15.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:06:15.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2023-11-02T09:06:15.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2023-11-02T09:06:15.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2023-11-02T09:06:15.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2023-11-02T09:06:15.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2023-11-02T09:06:15.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:06:15.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2023-11-02T09:06:15.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:06:15.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:06:15.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:06:15.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:06:15.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:06:15.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:06:15.749 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:06:15.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:06:15.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:06:15.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:06:15.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:06:15.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:06:15.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:06:15.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:06:15.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:06:15.751 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:06:15.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:06:15.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:06:15.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2023-11-02T09:06:15.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:06:15.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:06:15.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:06:15.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.212321 2023-11-02T09:06:15.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:06:15.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:06:15.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2023-11-02T09:06:15.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T09:06:15.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:06:15.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:06:15.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.212321 2023-11-02T09:06:15.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T09:06:15.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-11-02T09:06:15.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T09:06:15.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2023-11-02T09:06:15.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_default_adjustment td/osd-config 2023-11-02T09:06:15.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin 2023-11-02T09:06:15.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: a= 2023-11-02T09:06:15.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin default 2023-11-02T09:06:15.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: b=default 2023-11-02T09:06:15.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin arg 2023-11-02T09:06:15.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: c=arg 2023-11-02T09:06:15.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:81: TEST_default_adjustment: '[' '' '!=' default ']' 2023-11-02T09:06:15.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:82: TEST_default_adjustment: '[' default = default ']' 2023-11-02T09:06:15.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:83: TEST_default_adjustment: '[' arg = arg ']' 2023-11-02T09:06:15.807 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file 2023-11-02T09:06:15.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: a=true 2023-11-02T09:06:15.822 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false 2023-11-02T09:06:15.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: b=false 2023-11-02T09:06:15.841 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false --log-to-file 2023-11-02T09:06:15.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: c=true 2023-11-02T09:06:15.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:88: TEST_default_adjustment: '[' true = true ']' 2023-11-02T09:06:15.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:89: TEST_default_adjustment: '[' false = false ']' 2023-11-02T09:06:15.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:90: TEST_default_adjustment: '[' true = true ']' 2023-11-02T09:06:15.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2023-11-02T09:06:15.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2023-11-02T09:06:15.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:06:15.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2023-11-02T09:06:15.861 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:06:15.861 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:06:15.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:06:15.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:06:15.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:06:15.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:06:15.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:06:15.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:06:15.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:06:15.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:06:15.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:06:15.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:06:15.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:06:15.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:06:15.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:06:15.867 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:06:15.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:06:15.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:06:15.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2023-11-02T09:06:15.870 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:06:15.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:06:15.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:06:15.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.212321 2023-11-02T09:06:15.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:06:15.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:06:15.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2023-11-02T09:06:15.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-config 0 2023-11-02T09:06:15.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2023-11-02T09:06:15.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-11-02T09:06:15.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2023-11-02T09:06:15.873 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:06:15.873 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:06:15.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:06:15.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:06:15.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:06:15.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:06:15.875 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:06:15.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:06:15.879 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:06:15.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:06:15.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:06:15.879 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:06:15.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:06:15.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:06:15.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:06:15.880 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:06:15.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:06:15.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-11-02T09:06:15.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2023-11-02T09:06:15.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:06:15.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:06:15.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212321 2023-11-02T09:06:15.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.212321 2023-11-02T09:06:15.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:06:15.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:06:15.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2023-11-02T09:06:15.883 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-02T09:06:15.883 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-02T09:06:15.945 INFO:tasks.workunit:Running workunit osd/osd-copy-from.sh... 2023-11-02T09:06:15.946 DEBUG:teuthology.orchestra.run.smithi116:workunit test osd/osd-copy-from.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh 2023-11-02T09:06:16.006 INFO:tasks.workunit.client.0.smithi116.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-11-02T09:06:16.011 INFO:tasks.workunit.client.0.smithi116.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-11-02T09:06:16.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T09:06:16.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T09:06:16.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2023-11-02T09:06:16.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2023-11-02T09:06:16.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2023-11-02T09:06:16.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2023-11-02T09:06:16.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2023-11-02T09:06:16.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2023-11-02T09:06:16.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-copy-from 2023-11-02T09:06:16.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:23: run: local dir=td/osd-copy-from 2023-11-02T09:06:16.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:24: run: shift 2023-11-02T09:06:16.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: export CEPH_MON=127.0.0.1:7111 2023-11-02T09:06:16.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: CEPH_MON=127.0.0.1:7111 2023-11-02T09:06:16.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:27: run: export CEPH_ARGS 2023-11-02T09:06:16.014 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: uuidgen 2023-11-02T09:06:16.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: CEPH_ARGS+='--fsid=fbae51ed-85fb-42a2-8ee8-987f7a76d9f7 --auth-supported=none ' 2023-11-02T09:06:16.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7111 ' 2023-11-02T09:06:16.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: set 2023-11-02T09:06:16.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-11-02T09:06:16.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: local funcs=TEST_copy_from 2023-11-02T09:06:16.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:32: run: for func in $funcs 2023-11-02T09:06:16.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:33: run: setup td/osd-copy-from 2023-11-02T09:06:16.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-copy-from 2023-11-02T09:06:16.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-copy-from 2023-11-02T09:06:16.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2023-11-02T09:06:16.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:06:16.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2023-11-02T09:06:16.019 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:06:16.019 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:06:16.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:06:16.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:06:16.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:06:16.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:06:16.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:06:16.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:06:16.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:06:16.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:06:16.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:06:16.021 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:06:16.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:06:16.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:06:16.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:06:16.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:06:16.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:06:16.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:06:16.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2023-11-02T09:06:16.026 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:06:16.026 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:06:16.027 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.216422 2023-11-02T09:06:16.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.216422 2023-11-02T09:06:16.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:06:16.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:06:16.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-copy-from 2023-11-02T09:06:16.028 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T09:06:16.028 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:06:16.028 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.216422 2023-11-02T09:06:16.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.216422 2023-11-02T09:06:16.028 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T09:06:16.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-11-02T09:06:16.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-11-02T09:06:16.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T09:06:16.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-copy-from 1' TERM HUP INT 2023-11-02T09:06:16.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:34: run: TEST_copy_from td/osd-copy-from 2023-11-02T09:06:16.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:40: TEST_copy_from: local dir=td/osd-copy-from 2023-11-02T09:06:16.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:42: TEST_copy_from: run_mon td/osd-copy-from a 2023-11-02T09:06:16.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-copy-from 2023-11-02T09:06:16.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T09:06:16.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T09:06:16.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T09:06:16.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-copy-from/a 2023-11-02T09:06:16.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-copy-from/a --run-dir=td/osd-copy-from 2023-11-02T09:06:16.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T09:06:16.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:06:16.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:06:16.341 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:06:16.341 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:06:16.341 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.216422 2023-11-02T09:06:16.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.216422/$cluster-$name.asok' 2023-11-02T09:06:16.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-copy-from/a '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.216422/$cluster-$name.asok' --mon-cluster-log-file=td/osd-copy-from/log --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T09:06:16.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T09:06:16.532 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T09:06:16.532 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:06:16.533 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:06:16.533 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T09:06:16.533 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T09:06:16.533 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:06:16.534 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:06:16.534 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:06:16.534 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:06:16.534 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:06:16.535 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.216422 2023-11-02T09:06:16.535 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.216422/ceph-mon.a.asok 2023-11-02T09:06:16.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:06:16.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.216422/ceph-mon.a.asok config get fsid 2023-11-02T09:06:16.611 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T09:06:16.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:06:16.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:06:16.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T09:06:16.613 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T09:06:16.613 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:06:16.613 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:06:16.613 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:06:16.613 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:06:16.614 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:06:16.614 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.216422 2023-11-02T09:06:16.614 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.216422/ceph-mon.a.asok 2023-11-02T09:06:16.614 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:06:16.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.216422/ceph-mon.a.asok config get mon_host 2023-11-02T09:06:16.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:43: TEST_copy_from: run_mgr td/osd-copy-from x 2023-11-02T09:06:16.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-copy-from 2023-11-02T09:06:16.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T09:06:16.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T09:06:16.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T09:06:16.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-copy-from/x 2023-11-02T09:06:16.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T09:06:17.537 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T09:06:17.537 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:06:17.537 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:06:17.537 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:06:17.538 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:06:17.538 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.216422 2023-11-02T09:06:17.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.216422/$cluster-$name.asok' 2023-11-02T09:06:17.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:06:17.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-copy-from/x '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.216422/$cluster-$name.asok' --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:06:17.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:44: TEST_copy_from: run_osd td/osd-copy-from 0 2023-11-02T09:06:17.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2023-11-02T09:06:17.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:06:17.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T09:06:17.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:06:17.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/0 2023-11-02T09:06:17.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fbae51ed-85fb-42a2-8ee8-987f7a76d9f7 --auth-supported=none --mon-host=127.0.0.1:7111 ' 2023-11-02T09:06:17.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:06:17.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:06:17.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:06:17.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/0' 2023-11-02T09:06:17.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/0/journal' 2023-11-02T09:06:17.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:06:17.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:06:17.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2023-11-02T09:06:17.570 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:06:17.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:06:17.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:06:17.573 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:06:17.573 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:06:17.574 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.216422 2023-11-02T09:06:17.574 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.216422/$cluster-$name.asok' 2023-11-02T09:06:17.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.216422/$cluster-$name.asok' 2023-11-02T09:06:17.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:06:17.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:06:17.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:06:17.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2023-11-02T09:06:17.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2023-11-02T09:06:17.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:06:17.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:06:17.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:06:17.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:06:17.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:06:17.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:06:17.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-copy-from/0 2023-11-02T09:06:17.577 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:06:17.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5a7c5f0c-7ee0-428c-9b38-49ac70abc62c 2023-11-02T09:06:17.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5a7c5f0c-7ee0-428c-9b38-49ac70abc62c' 2023-11-02T09:06:17.578 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 5a7c5f0c-7ee0-428c-9b38-49ac70abc62c 2023-11-02T09:06:17.578 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:06:17.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCJZkNldTpvIhAADl04Y8X9JdxWf2qz9j96zQ== 2023-11-02T09:06:17.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCJZkNldTpvIhAADl04Y8X9JdxWf2qz9j96zQ=="}' 2023-11-02T09:06:17.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5a7c5f0c-7ee0-428c-9b38-49ac70abc62c -i td/osd-copy-from/0/new.json 2023-11-02T09:06:17.939 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:06:17.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-copy-from/0/new.json 2023-11-02T09:06:17.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fbae51ed-85fb-42a2-8ee8-987f7a76d9f7 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.216422/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCJZkNldTpvIhAADl04Y8X9JdxWf2qz9j96zQ== --osd-uuid 5a7c5f0c-7ee0-428c-9b38-49ac70abc62c 2023-11-02T09:06:17.979 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:17.972+0000 7f6b75623500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:06:17.979 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:17.974+0000 7f6b75623500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:06:17.979 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:17.975+0000 7f6b75623500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:06:17.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:17.976+0000 7f6b75623500 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2023-11-02T09:06:17.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:17.976+0000 7f6b75623500 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2023-11-02T09:06:17.980 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:17.976+0000 7f6b75623500 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2023-11-02T09:06:18.021 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:18.019+0000 7f6b75623500 -1 bluestore(td/osd-copy-from/0) _read_fsid unparsable uuid 2023-11-02T09:06:23.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-copy-from/0/keyring 2023-11-02T09:06:23.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:06:23.253 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T09:06:23.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T09:06:23.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-copy-from/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:06:23.780 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T09:06:23.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T09:06:23.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fbae51ed-85fb-42a2-8ee8-987f7a76d9f7 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.216422/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:06:23.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:06:23.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:06:23.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:06:23.805 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:23.802+0000 7fe31ee69500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:06:23.807 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:23.804+0000 7fe31ee69500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:06:23.808 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:23.805+0000 7fe31ee69500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:06:24.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T09:06:24.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:06:24.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T09:06:24.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:06:24.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:06:24.191 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:06:24.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:24.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:06:24.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:24.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:24.323 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:24.321+0000 7fe31ee69500 -1 Falling back to public interface 2023-11-02T09:06:24.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:25.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:25.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:25.601 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:06:25.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:06:25.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:25.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:26.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:26.611 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:26.608+0000 7fe31ee69500 -1 osd.0 0 log_to_monitors true 2023-11-02T09:06:27.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:27.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:27.011 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:06:27.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:06:27.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:27.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:27.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:28.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:28.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:28.379 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:06:28.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:06:28.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:28.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:28.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:29.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:29.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:29.787 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:06:29.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:06:29.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:29.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:30.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:31.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:31.318 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:06:31.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:31.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:06:31.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:31.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:31.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:32.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:32.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:32.789 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:06:32.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:06:32.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:32.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:33.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:34.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:34.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:34.314 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:06:34.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:06:34.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:34.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:34.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:35.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:35.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:35.725 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:06:35.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:06:35.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:35.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:36.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:37.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:37.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:37.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:06:37.136 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:06:37.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:37.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:37.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:38.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:38.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:38.621 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:06:38.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:06:38.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:38.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:39.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:40.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:40.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:40.192 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:06:40.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:06:40.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:40.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:40.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:41.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:41.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:41.791 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:06:41.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:06:41.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:41.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:42.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:43.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:43.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:43.493 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:06:43.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:06:43.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:43.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:43.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:44.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:44.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:44.878 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:06:44.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:06:44.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:44.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:45.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:46.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:46.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:46.529 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:06:46.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:06:46.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:46.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:47.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:48.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:48.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:48.685 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:06:48.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:06:48.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:48.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:06:49.381 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/217506,v1:127.0.0.1:6803/217506] [v2:127.0.0.1:6804/217506,v1:127.0.0.1:6805/217506] exists,up 5a7c5f0c-7ee0-428c-9b38-49ac70abc62c 2023-11-02T09:06:49.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:06:49.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:06:49.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:06:49.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:45: TEST_copy_from: run_osd td/osd-copy-from 1 2023-11-02T09:06:49.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2023-11-02T09:06:49.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:06:49.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-11-02T09:06:49.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:06:49.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/1 2023-11-02T09:06:49.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fbae51ed-85fb-42a2-8ee8-987f7a76d9f7 --auth-supported=none --mon-host=127.0.0.1:7111 ' 2023-11-02T09:06:49.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:06:49.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:06:49.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:06:49.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/1' 2023-11-02T09:06:49.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/1/journal' 2023-11-02T09:06:49.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:06:49.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:06:49.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2023-11-02T09:06:49.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:06:49.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:06:49.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:06:49.390 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:06:49.390 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:06:49.390 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.216422 2023-11-02T09:06:49.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.216422/$cluster-$name.asok' 2023-11-02T09:06:49.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.216422/$cluster-$name.asok' 2023-11-02T09:06:49.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:06:49.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:06:49.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:06:49.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2023-11-02T09:06:49.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2023-11-02T09:06:49.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:06:49.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:06:49.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:06:49.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:06:49.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:06:49.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:06:49.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-copy-from/1 2023-11-02T09:06:49.393 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:06:49.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9d700f35-a317-4c9c-b181-4862f378351a 2023-11-02T09:06:49.394 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 9d700f35-a317-4c9c-b181-4862f378351a 2023-11-02T09:06:49.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9d700f35-a317-4c9c-b181-4862f378351a' 2023-11-02T09:06:49.394 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:06:49.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCpZkNlzwqyFxAAnkSBFeEWolIQ2eTSwzjXAA== 2023-11-02T09:06:49.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCpZkNlzwqyFxAAnkSBFeEWolIQ2eTSwzjXAA=="}' 2023-11-02T09:06:49.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9d700f35-a317-4c9c-b181-4862f378351a -i td/osd-copy-from/1/new.json 2023-11-02T09:06:50.008 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:06:50.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-copy-from/1/new.json 2023-11-02T09:06:50.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=fbae51ed-85fb-42a2-8ee8-987f7a76d9f7 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.216422/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCpZkNlzwqyFxAAnkSBFeEWolIQ2eTSwzjXAA== --osd-uuid 9d700f35-a317-4c9c-b181-4862f378351a 2023-11-02T09:06:50.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:50.043+0000 7f6e3c0af500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:06:50.047 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:50.044+0000 7f6e3c0af500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:06:50.048 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:50.045+0000 7f6e3c0af500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:06:50.049 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:50.046+0000 7f6e3c0af500 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2023-11-02T09:06:50.049 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:50.046+0000 7f6e3c0af500 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2023-11-02T09:06:50.049 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:50.046+0000 7f6e3c0af500 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2023-11-02T09:06:50.157 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:50.154+0000 7f6e3c0af500 -1 bluestore(td/osd-copy-from/1) _read_fsid unparsable uuid 2023-11-02T09:06:55.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-copy-from/1/keyring 2023-11-02T09:06:55.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:06:55.573 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2023-11-02T09:06:55.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-11-02T09:06:55.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-copy-from/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:06:56.098 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T09:06:56.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-11-02T09:06:56.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=fbae51ed-85fb-42a2-8ee8-987f7a76d9f7 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.216422/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:06:56.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:06:56.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:06:56.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:06:56.121 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:56.118+0000 7f97ab9ad500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:06:56.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:56.142+0000 7f97ab9ad500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:06:56.146 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:56.144+0000 7f97ab9ad500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:06:56.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-11-02T09:06:56.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:06:56.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T09:06:56.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:06:56.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:06:56.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:56.552 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:06:56.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:06:56.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:56.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:06:56.690 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:56.687+0000 7f97ab9ad500 -1 Falling back to public interface 2023-11-02T09:06:56.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:57.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:57.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:57.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:06:57.952 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:06:57.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:57.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:06:58.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:06:58.955 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:06:58.952+0000 7f97ab9ad500 -1 osd.1 0 log_to_monitors true 2023-11-02T09:06:59.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:06:59.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:06:59.371 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:06:59.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:06:59.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:06:59.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:06:59.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:00.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:00.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:00.842 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:07:00.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:07:00.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:00.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:07:01.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:02.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:02.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:02.239 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:07:02.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:07:02.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:02.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:07:02.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:03.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:03.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:03.746 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:07:03.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:07:03.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:03.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:07:04.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:05.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:05.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:05.185 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:07:05.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:07:05.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:05.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:07:05.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:06.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:06.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:06.697 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:07:06.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:07:06.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:06.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:07:07.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:08.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:08.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:08.107 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:07:08.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:07:08.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:08.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:07:08.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:09.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:09.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:09.500 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:07:09.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:07:09.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:09.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:07:09.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:10.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:10.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:10.991 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:07:10.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:07:10.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:10.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:07:11.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:12.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:12.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:12.443 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:07:12.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:07:12.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:12.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:07:13.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:14.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:14.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:14.275 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:07:14.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:07:14.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:14.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:07:14.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:15.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:15.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:15.884 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:07:15.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:07:15.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:15.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:07:16.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:17.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:17.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:17.397 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:07:17.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:07:17.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:17.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:07:18.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:19.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:19.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:19.293 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:07:19.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:07:19.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:19.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:07:20.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:21.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:21.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:21.061 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:07:21.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:07:21.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:21.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:07:21.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:22.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:22.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:22.769 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T09:07:22.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T09:07:22.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:22.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:07:23.204 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/219219,v1:127.0.0.1:6811/219219] [v2:127.0.0.1:6812/219219,v1:127.0.0.1:6813/219219] exists,up 9d700f35-a317-4c9c-b181-4862f378351a 2023-11-02T09:07:23.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:07:23.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:07:23.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:07:23.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:46: TEST_copy_from: create_rbd_pool 2023-11-02T09:07:23.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-11-02T09:07:23.592 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' does not exist 2023-11-02T09:07:23.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-11-02T09:07:23.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-11-02T09:07:24.439 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' already exists 2023-11-02T09:07:24.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:07:25.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-11-02T09:07:26.703 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: which rados 2023-11-02T09:07:26.703 INFO:tasks.workunit.client.0.smithi116.stderr:eenvironment:0: which: alias 2023-11-02T09:07:26.703 INFO:tasks.workunit.client.0.smithi116.stderr:eenvironment:1: which: eval declare -f 2023-11-02T09:07:26.704 INFO:tasks.workunit.client.0.smithi116.stderr:eeenvironment:1: which: declare -f 2023-11-02T09:07:26.704 INFO:tasks.workunit.client.0.smithi116.stderr:eenvironment:1: which: /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot rados 2023-11-02T09:07:26.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: rados -p rbd put foo /usr/bin/rados 2023-11-02T09:07:26.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:50: TEST_copy_from: rados -p rbd cp foo foo2 2023-11-02T09:07:27.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:51: TEST_copy_from: rados -p rbd stat foo2 2023-11-02T09:07:27.064 INFO:tasks.workunit.client.0.smithi116.stdout:rbd/foo2 mtime 2023-11-02T09:07:26.000000+0000, size 604496 2023-11-02T09:07:27.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:54: TEST_copy_from: ceph tell 'osd.*' injectargs -- --osd-debug-inject-copyfrom-error 2023-11-02T09:07:27.174 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0: {} 2023-11-02T09:07:27.202 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1: {} 2023-11-02T09:07:27.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:55: TEST_copy_from: rados -p rbd cp foo foo3 2023-11-02T09:07:27.244 INFO:tasks.workunit.client.0.smithi116.stderr:error copying rbd/foo => rbd/foo3: (5) Input/output error 2023-11-02T09:07:27.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:56: TEST_copy_from: rados -p rbd stat foo3 2023-11-02T09:07:27.298 INFO:tasks.workunit.client.0.smithi116.stderr: error stat-ing rbd/foo3: (2) No such file or directory 2023-11-02T09:07:27.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:59: TEST_copy_from: ceph tell 'osd.*' injectargs -- --no-osd-debug-inject-copyfrom-error 2023-11-02T09:07:27.402 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0: {} 2023-11-02T09:07:27.430 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1: {} 2023-11-02T09:07:27.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:60: TEST_copy_from: rados -p rbd cp foo foo3 2023-11-02T09:07:27.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:61: TEST_copy_from: rados -p rbd stat foo3 2023-11-02T09:07:27.681 INFO:tasks.workunit.client.0.smithi116.stdout:rbd/foo3 mtime 2023-11-02T09:07:26.000000+0000, size 604496 2023-11-02T09:07:27.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:35: run: teardown td/osd-copy-from 2023-11-02T09:07:27.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2023-11-02T09:07:27.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:07:27.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2023-11-02T09:07:27.684 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:07:27.685 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:07:27.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:07:27.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:07:27.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:07:27.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:07:27.790 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:07:27.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:07:27.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:07:27.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:07:27.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:07:27.792 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:07:27.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:07:27.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:07:27.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:07:27.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:07:27.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:07:27.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:07:27.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2023-11-02T09:07:27.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:07:27.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:07:27.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.216422 2023-11-02T09:07:27.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.216422 2023-11-02T09:07:27.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:07:27.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:07:27.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2023-11-02T09:07:27.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-copy-from 0 2023-11-02T09:07:27.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2023-11-02T09:07:27.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-11-02T09:07:27.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2023-11-02T09:07:27.807 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:07:27.807 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:07:27.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:07:27.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:07:27.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:07:27.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:07:27.810 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:07:27.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:07:27.810 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:07:27.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:07:27.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:07:27.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:07:27.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:07:27.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:07:27.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:07:27.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:07:27.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:07:27.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-11-02T09:07:27.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2023-11-02T09:07:27.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:07:27.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:07:27.814 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.216422 2023-11-02T09:07:27.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.216422 2023-11-02T09:07:27.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:07:27.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:07:27.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2023-11-02T09:07:27.816 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-02T09:07:27.816 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-02T09:07:27.882 INFO:tasks.workunit:Running workunit osd/osd-dup.sh... 2023-11-02T09:07:27.883 DEBUG:teuthology.orchestra.run.smithi116:workunit test osd/osd-dup.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh 2023-11-02T09:07:27.943 INFO:tasks.workunit.client.0.smithi116.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-11-02T09:07:27.949 INFO:tasks.workunit.client.0.smithi116.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-11-02T09:07:27.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T09:07:27.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T09:07:27.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2023-11-02T09:07:27.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2023-11-02T09:07:27.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2023-11-02T09:07:27.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2023-11-02T09:07:27.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2023-11-02T09:07:27.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2023-11-02T09:07:27.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-dup 2023-11-02T09:07:27.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:8: run: local dir=td/osd-dup 2023-11-02T09:07:27.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:9: run: shift 2023-11-02T09:07:27.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: export CEPH_MON=127.0.0.1:7146 2023-11-02T09:07:27.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: CEPH_MON=127.0.0.1:7146 2023-11-02T09:07:27.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:12: run: export CEPH_ARGS 2023-11-02T09:07:27.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: uuidgen 2023-11-02T09:07:27.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: CEPH_ARGS+='--fsid=9883d25f-b628-430d-8798-04d335aac855 --auth-supported=none ' 2023-11-02T09:07:27.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2023-11-02T09:07:27.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:16: run: CEPH_ARGS+='--filestore_wbthrottle_xfs_ios_hard_limit=900 ' 2023-11-02T09:07:27.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:17: run: CEPH_ARGS+='--filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2023-11-02T09:07:27.953 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: set 2023-11-02T09:07:27.953 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-11-02T09:07:27.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: local funcs= 2023-11-02T09:07:27.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2023-11-02T09:07:27.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-dup 0 2023-11-02T09:07:27.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2023-11-02T09:07:27.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-11-02T09:07:27.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2023-11-02T09:07:27.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:07:27.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:07:27.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:07:27.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:07:27.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:07:27.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:07:27.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:07:27.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:07:27.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:07:27.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:07:27.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:07:27.960 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:07:27.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:07:27.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:07:27.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:07:27.960 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:07:27.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:07:27.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-11-02T09:07:27.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2023-11-02T09:07:27.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:07:27.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:07:27.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.220466 2023-11-02T09:07:27.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.220466 2023-11-02T09:07:27.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:07:27.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:07:27.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2023-11-02T09:07:27.964 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-02T09:07:27.965 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-02T09:07:28.029 INFO:tasks.workunit:Running workunit osd/osd-fast-mark-down.sh... 2023-11-02T09:07:28.029 DEBUG:teuthology.orchestra.run.smithi116:workunit test osd/osd-fast-mark-down.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh 2023-11-02T09:07:28.089 INFO:tasks.workunit.client.0.smithi116.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-11-02T09:07:28.093 INFO:tasks.workunit.client.0.smithi116.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-11-02T09:07:28.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T09:07:28.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T09:07:28.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2023-11-02T09:07:28.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2023-11-02T09:07:28.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2023-11-02T09:07:28.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2023-11-02T09:07:28.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2023-11-02T09:07:28.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2023-11-02T09:07:28.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-fast-mark-down 2023-11-02T09:07:28.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:23: run: local dir=td/osd-fast-mark-down 2023-11-02T09:07:28.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:24: run: shift 2023-11-02T09:07:28.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:25: run: rm -f 'td/osd-fast-mark-down/*.pid' 2023-11-02T09:07:28.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: export CEPH_MON=127.0.0.1:7126 2023-11-02T09:07:28.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: CEPH_MON=127.0.0.1:7126 2023-11-02T09:07:28.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:27: run: export CEPH_ARGS 2023-11-02T09:07:28.098 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: uuidgen 2023-11-02T09:07:28.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: CEPH_ARGS+='--fsid=871ba12a-f05c-4576-8e0f-2a15553d0ce9 --auth-supported=none ' 2023-11-02T09:07:28.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:30: run: OLD_ARGS='--fsid=871ba12a-f05c-4576-8e0f-2a15553d0ce9 --auth-supported=none ' 2023-11-02T09:07:28.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:31: run: CEPH_ARGS+='--osd-fast-fail-on-connection-refused=false ' 2023-11-02T09:07:28.099 INFO:tasks.workunit.client.0.smithi116.stdout:Ensuring old behavior is there... 2023-11-02T09:07:28.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:32: run: echo 'Ensuring old behavior is there...' 2023-11-02T09:07:28.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:33: run: test_fast_kill td/osd-fast-mark-down 2023-11-02T09:07:28.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2023-11-02T09:07:28.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2023-11-02T09:07:28.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2023-11-02T09:07:28.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2023-11-02T09:07:28.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:07:28.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2023-11-02T09:07:28.104 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:07:28.105 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:07:28.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:07:28.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:07:28.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:07:28.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:07:28.106 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:07:28.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:07:28.106 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:07:28.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:07:28.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:07:28.107 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:07:28.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:07:28.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:07:28.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:07:28.108 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:07:28.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:07:28.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:07:28.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2023-11-02T09:07:28.108 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:07:28.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:07:28.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.220526 2023-11-02T09:07:28.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.220526 2023-11-02T09:07:28.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:07:28.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:07:28.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2023-11-02T09:07:28.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T09:07:28.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:07:28.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.220526 2023-11-02T09:07:28.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.220526 2023-11-02T09:07:28.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T09:07:28.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-11-02T09:07:28.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-11-02T09:07:28.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T09:07:28.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2023-11-02T09:07:28.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2023-11-02T09:07:28.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2023-11-02T09:07:28.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T09:07:28.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T09:07:28.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T09:07:28.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2023-11-02T09:07:28.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2023-11-02T09:07:28.126 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:07:28.124+0000 7fd763848b00 -1 failed for service _ceph-mon._tcp 2023-11-02T09:07:28.127 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:07:28.124+0000 7fd763848b00 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2023-11-02T09:07:28.127 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-11-02T09:07:28.127 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:07:28.124+0000 7fd763848b00 -1 ceph-mon: warning: no initial monitors; must use admin socket to feed hints 2023-11-02T09:07:28.476 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T09:07:28.476 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:07:28.477 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:07:28.477 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:07:28.477 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:07:28.477 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.220526 2023-11-02T09:07:28.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.220526/$cluster-$name.asok' 2023-11-02T09:07:28.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.220526/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2023-11-02T09:07:28.648 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:07:28.645+0000 7fadf6e0db00 -1 failed for service _ceph-mon._tcp 2023-11-02T09:07:28.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:07:28.645+0000 7fadf6e0db00 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2023-11-02T09:07:28.649 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-11-02T09:07:28.649 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:07:28.645+0000 7fadf6e0db00 -1 ceph-mon: error generating initial monmap: (2) No such file or directory 2023-11-02T09:07:28.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: return 1 2023-11-02T09:07:28.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: return 1 2023-11-02T09:07:28.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:35: run: CEPH_ARGS='--fsid=871ba12a-f05c-4576-8e0f-2a15553d0ce9 --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2023-11-02T09:07:28.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:36: run: OLD_ARGS='--fsid=871ba12a-f05c-4576-8e0f-2a15553d0ce9 --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2023-11-02T09:07:28.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:38: run: CEPH_ARGS='--fsid=871ba12a-f05c-4576-8e0f-2a15553d0ce9 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2023-11-02T09:07:28.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:39: run: echo 'Testing async msgr...' 2023-11-02T09:07:28.653 INFO:tasks.workunit.client.0.smithi116.stdout:Testing async msgr... 2023-11-02T09:07:28.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:40: run: test_fast_kill td/osd-fast-mark-down 2023-11-02T09:07:28.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2023-11-02T09:07:28.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2023-11-02T09:07:28.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2023-11-02T09:07:28.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2023-11-02T09:07:28.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:07:28.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2023-11-02T09:07:28.656 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:07:28.657 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:07:28.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:07:28.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:07:28.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:07:28.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:07:28.658 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:07:28.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:07:28.658 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:07:28.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:07:28.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:07:28.660 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:07:28.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:07:28.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:07:28.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:07:28.661 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:07:28.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:07:28.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:07:28.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2023-11-02T09:07:28.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:07:28.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:07:28.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.220526 2023-11-02T09:07:28.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.220526 2023-11-02T09:07:28.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:07:28.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:07:28.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2023-11-02T09:07:28.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T09:07:28.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:07:28.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.220526 2023-11-02T09:07:28.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.220526 2023-11-02T09:07:28.668 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T09:07:28.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-11-02T09:07:28.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T09:07:28.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2023-11-02T09:07:28.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2023-11-02T09:07:28.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2023-11-02T09:07:28.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T09:07:28.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T09:07:28.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T09:07:28.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2023-11-02T09:07:28.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2023-11-02T09:07:29.046 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T09:07:29.047 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:07:29.047 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:07:29.047 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:07:29.047 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:07:29.048 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.220526 2023-11-02T09:07:29.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.220526/$cluster-$name.asok' 2023-11-02T09:07:29.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.220526/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2023-11-02T09:07:29.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T09:07:29.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T09:07:29.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:07:29.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:07:29.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T09:07:29.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T09:07:29.217 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:07:29.217 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:07:29.217 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:07:29.217 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:07:29.217 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:07:29.218 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.220526 2023-11-02T09:07:29.218 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.220526/ceph-mon.a.asok 2023-11-02T09:07:29.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:07:29.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.220526/ceph-mon.a.asok config get fsid 2023-11-02T09:07:29.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T09:07:29.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:07:29.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:07:29.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T09:07:29.296 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T09:07:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:07:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:07:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:07:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:07:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:07:29.297 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.220526 2023-11-02T09:07:29.297 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.220526/ceph-mon.a.asok 2023-11-02T09:07:29.297 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:07:29.297 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.220526/ceph-mon.a.asok config get mon_host 2023-11-02T09:07:29.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:50: test_fast_kill: run_mgr td/osd-fast-mark-down x 2023-11-02T09:07:29.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-fast-mark-down 2023-11-02T09:07:29.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T09:07:29.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T09:07:29.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T09:07:29.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-fast-mark-down/x 2023-11-02T09:07:29.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T09:07:30.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T09:07:30.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:07:30.230 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:07:30.230 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:07:30.230 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:07:30.230 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.220526 2023-11-02T09:07:30.230 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.220526/$cluster-$name.asok' 2023-11-02T09:07:30.231 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:07:30.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-fast-mark-down/x '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.220526/$cluster-$name.asok' --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:07:30.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2023-11-02T09:07:30.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 0 2023-11-02T09:07:30.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2023-11-02T09:07:30.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:07:30.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T09:07:30.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:07:30.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/0 2023-11-02T09:07:30.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=871ba12a-f05c-4576-8e0f-2a15553d0ce9 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2023-11-02T09:07:30.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:07:30.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:07:30.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:07:30.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/0' 2023-11-02T09:07:30.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/0/journal' 2023-11-02T09:07:30.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:07:30.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:07:30.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2023-11-02T09:07:30.263 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:07:30.263 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:07:30.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:07:30.264 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:07:30.264 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:07:30.264 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.220526 2023-11-02T09:07:30.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.220526/$cluster-$name.asok' 2023-11-02T09:07:30.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.220526/$cluster-$name.asok' 2023-11-02T09:07:30.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:07:30.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:07:30.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:07:30.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2023-11-02T09:07:30.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2023-11-02T09:07:30.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:07:30.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:07:30.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:07:30.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:07:30.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:07:30.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:07:30.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/0 2023-11-02T09:07:30.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:07:30.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=30218ccb-727a-4bef-b74c-3fd356ae6655 2023-11-02T09:07:30.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 30218ccb-727a-4bef-b74c-3fd356ae6655' 2023-11-02T09:07:30.268 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 30218ccb-727a-4bef-b74c-3fd356ae6655 2023-11-02T09:07:30.269 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:07:30.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDSZkNl+1F1EBAAqj0BflDTPRG3h+FiZ5ceYA== 2023-11-02T09:07:30.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDSZkNl+1F1EBAAqj0BflDTPRG3h+FiZ5ceYA=="}' 2023-11-02T09:07:30.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 30218ccb-727a-4bef-b74c-3fd356ae6655 -i td/osd-fast-mark-down/0/new.json 2023-11-02T09:07:30.614 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:07:30.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/0/new.json 2023-11-02T09:07:30.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=871ba12a-f05c-4576-8e0f-2a15553d0ce9 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.220526/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDSZkNl+1F1EBAAqj0BflDTPRG3h+FiZ5ceYA== --osd-uuid 30218ccb-727a-4bef-b74c-3fd356ae6655 2023-11-02T09:07:30.654 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:07:30.651+0000 7f502321f500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:07:30.656 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:07:30.653+0000 7f502321f500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:07:30.657 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:07:30.654+0000 7f502321f500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:07:30.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:07:30.654+0000 7f502321f500 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2023-11-02T09:07:30.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:07:30.654+0000 7f502321f500 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2023-11-02T09:07:30.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:07:30.654+0000 7f502321f500 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2023-11-02T09:07:30.696 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:07:30.694+0000 7f502321f500 -1 bluestore(td/osd-fast-mark-down/0) _read_fsid unparsable uuid 2023-11-02T09:07:36.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/0/keyring 2023-11-02T09:07:36.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:07:36.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T09:07:36.121 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T09:07:36.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:07:36.639 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T09:07:36.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T09:07:36.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=871ba12a-f05c-4576-8e0f-2a15553d0ce9 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.220526/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:07:36.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:07:36.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:07:36.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:07:36.666 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:07:36.663+0000 7f248ee12500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:07:36.686 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:07:36.683+0000 7f248ee12500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:07:36.687 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:07:36.684+0000 7f248ee12500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:07:37.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T09:07:37.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:07:37.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T09:07:37.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:07:37.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:07:37.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:37.109 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:07:37.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:07:37.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:37.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:07:37.216 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:07:37.214+0000 7f248ee12500 -1 Falling back to public interface 2023-11-02T09:07:37.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:38.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:38.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:38.526 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:07:38.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:07:38.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:38.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:07:38.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:39.528 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:07:39.526+0000 7f248ee12500 -1 osd.0 0 log_to_monitors true 2023-11-02T09:07:39.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:39.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:39.952 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:07:39.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:07:39.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:39.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:07:40.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:41.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:41.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:41.464 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:07:41.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:07:41.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:41.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:07:41.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:42.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:42.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:42.887 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:07:42.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:07:42.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:42.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:07:43.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:44.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:44.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:44.418 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:07:44.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:07:44.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:44.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:07:44.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:45.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:45.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:45.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:07:45.816 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:07:45.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:45.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:07:46.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:47.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:47.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:47.337 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:07:47.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:07:47.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:47.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:07:47.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:48.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:48.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:48.733 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:07:48.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:07:48.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:48.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:07:49.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:50.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:50.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:50.207 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:07:50.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:07:50.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:50.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:07:50.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:51.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:51.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:51.619 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:07:51.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:07:51.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:51.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:07:52.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:53.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:53.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:53.626 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:07:53.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:07:53.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:53.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:07:54.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:55.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:55.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:55.340 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:07:55.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:07:55.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:55.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:07:55.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:56.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:56.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:56.955 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:07:56.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:07:56.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:56.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:07:57.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:07:58.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:07:58.794 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:07:58.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:07:58.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:07:58.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:07:58.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:07:59.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:00.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:00.338 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:08:00.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:00.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:08:00.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:00.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:08:01.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:02.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:02.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:02.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:08:02.450 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:08:02.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:02.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:08:02.846 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/221675,v1:127.0.0.1:6803/221675] [v2:127.0.0.1:6804/221675,v1:127.0.0.1:6805/221675] exists,up 30218ccb-727a-4bef-b74c-3fd356ae6655 2023-11-02T09:08:02.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:08:02.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:08:02.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:08:02.851 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.0.pid 2023-11-02T09:08:02.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=221675 2023-11-02T09:08:02.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2023-11-02T09:08:02.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 1 2023-11-02T09:08:02.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2023-11-02T09:08:02.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:08:02.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-11-02T09:08:02.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:08:02.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/1 2023-11-02T09:08:02.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=871ba12a-f05c-4576-8e0f-2a15553d0ce9 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2023-11-02T09:08:02.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:08:02.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:08:02.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:08:02.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/1' 2023-11-02T09:08:02.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/1/journal' 2023-11-02T09:08:02.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:08:02.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:08:02.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2023-11-02T09:08:02.855 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:08:02.856 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:08:02.856 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:08:02.856 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:08:02.856 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:08:02.856 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.220526 2023-11-02T09:08:02.857 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.220526/$cluster-$name.asok' 2023-11-02T09:08:02.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.220526/$cluster-$name.asok' 2023-11-02T09:08:02.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:08:02.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:08:02.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:08:02.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2023-11-02T09:08:02.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2023-11-02T09:08:02.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:08:02.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:08:02.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:08:02.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:08:02.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:08:02.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:08:02.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/1 2023-11-02T09:08:02.859 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:08:02.860 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 89537d62-e35d-4ca3-8ceb-7f8c2734a6e6 2023-11-02T09:08:02.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=89537d62-e35d-4ca3-8ceb-7f8c2734a6e6 2023-11-02T09:08:02.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 89537d62-e35d-4ca3-8ceb-7f8c2734a6e6' 2023-11-02T09:08:02.861 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:08:02.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDyZkNl2LunMxAAq6Nf2xlZvBa/Wczic+AtCA== 2023-11-02T09:08:02.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDyZkNl2LunMxAAq6Nf2xlZvBa/Wczic+AtCA=="}' 2023-11-02T09:08:02.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 89537d62-e35d-4ca3-8ceb-7f8c2734a6e6 -i td/osd-fast-mark-down/1/new.json 2023-11-02T09:08:03.340 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:08:03.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/1/new.json 2023-11-02T09:08:03.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=871ba12a-f05c-4576-8e0f-2a15553d0ce9 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.220526/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDyZkNl2LunMxAAq6Nf2xlZvBa/Wczic+AtCA== --osd-uuid 89537d62-e35d-4ca3-8ceb-7f8c2734a6e6 2023-11-02T09:08:03.380 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:03.377+0000 7f68a5c20500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:08:03.385 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:03.382+0000 7f68a5c20500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:08:03.387 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:03.383+0000 7f68a5c20500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:08:03.387 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:03.384+0000 7f68a5c20500 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2023-11-02T09:08:03.387 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:03.384+0000 7f68a5c20500 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2023-11-02T09:08:03.388 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:03.384+0000 7f68a5c20500 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2023-11-02T09:08:03.417 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:03.414+0000 7f68a5c20500 -1 bluestore(td/osd-fast-mark-down/1) _read_fsid unparsable uuid 2023-11-02T09:08:08.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/1/keyring 2023-11-02T09:08:08.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:08:08.872 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2023-11-02T09:08:08.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-11-02T09:08:08.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:08:09.391 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T09:08:09.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-11-02T09:08:09.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=871ba12a-f05c-4576-8e0f-2a15553d0ce9 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.220526/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:08:09.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:08:09.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:08:09.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:08:09.419 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:09.416+0000 7fa2b652b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:08:09.430 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:09.427+0000 7fa2b652b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:08:09.431 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:09.428+0000 7fa2b652b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:08:09.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-11-02T09:08:09.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:08:09.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T09:08:09.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:08:09.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:08:09.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:09.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:08:09.830 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:08:09.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:09.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:08:09.965 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:09.959+0000 7fa2b652b500 -1 Falling back to public interface 2023-11-02T09:08:10.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:11.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:11.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:11.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:08:11.230 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:08:11.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:11.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:08:11.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:12.256 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:12.253+0000 7fa2b652b500 -1 osd.1 0 log_to_monitors true 2023-11-02T09:08:12.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:12.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:12.647 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:08:12.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:08:12.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:12.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:08:13.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:14.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:14.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:14.061 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:08:14.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:08:14.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:14.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:08:14.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:15.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:15.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:15.576 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:08:15.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:08:15.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:15.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:08:16.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:17.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:17.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:17.018 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:08:17.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:08:17.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:17.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:08:17.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:18.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:18.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:18.470 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:08:18.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:08:18.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:18.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:08:19.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:20.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:20.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:20.051 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:08:20.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:08:20.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:20.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:08:20.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:21.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:21.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:21.466 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:08:21.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:08:21.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:21.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:08:21.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:22.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:22.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:22.847 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:08:22.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:08:22.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:22.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:08:23.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:24.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:24.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:24.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:08:24.360 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:08:24.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:24.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:08:24.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:25.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:25.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:25.899 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:08:25.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:08:25.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:25.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:08:26.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:27.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:27.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:27.724 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:08:27.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:08:27.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:27.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:08:28.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:29.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:29.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:29.462 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:08:29.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:08:29.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:29.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:08:30.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:31.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:31.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:31.462 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:08:31.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:08:31.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:31.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:08:31.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:32.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:32.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:32.942 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:08:32.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:08:32.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:32.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:08:34.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:35.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:35.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:35.080 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:08:35.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:08:35.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:35.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:08:35.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:36.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:36.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:36.535 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T09:08:36.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T09:08:36.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:36.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:08:36.944 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/223389,v1:127.0.0.1:6811/223389] [v2:127.0.0.1:6812/223389,v1:127.0.0.1:6813/223389] exists,up 89537d62-e35d-4ca3-8ceb-7f8c2734a6e6 2023-11-02T09:08:36.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:08:36.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:08:36.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:08:36.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.1.pid 2023-11-02T09:08:36.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=223389 2023-11-02T09:08:36.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2023-11-02T09:08:36.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 2 2023-11-02T09:08:36.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2023-11-02T09:08:36.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:08:36.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-11-02T09:08:36.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:08:36.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/2 2023-11-02T09:08:36.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=871ba12a-f05c-4576-8e0f-2a15553d0ce9 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2023-11-02T09:08:36.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:08:36.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:08:36.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:08:36.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/2' 2023-11-02T09:08:36.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/2/journal' 2023-11-02T09:08:36.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:08:36.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:08:36.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2023-11-02T09:08:36.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:08:36.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:08:36.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:08:36.955 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:08:36.957 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:08:36.958 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.220526 2023-11-02T09:08:36.958 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.220526/$cluster-$name.asok' 2023-11-02T09:08:36.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.220526/$cluster-$name.asok' 2023-11-02T09:08:36.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:08:36.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:08:36.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:08:36.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2023-11-02T09:08:36.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2023-11-02T09:08:36.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:08:36.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:08:36.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:08:36.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:08:36.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:08:36.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:08:36.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/2 2023-11-02T09:08:36.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:08:36.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=58073f00-433f-4cc7-b61e-00c1804d0032 2023-11-02T09:08:36.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 58073f00-433f-4cc7-b61e-00c1804d0032' 2023-11-02T09:08:36.962 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 58073f00-433f-4cc7-b61e-00c1804d0032 2023-11-02T09:08:36.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:08:36.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAUZ0NlxkqFORAA5oGoJ60zQN2GTTiWi7XMzw== 2023-11-02T09:08:36.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAUZ0NlxkqFORAA5oGoJ60zQN2GTTiWi7XMzw=="}' 2023-11-02T09:08:36.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 58073f00-433f-4cc7-b61e-00c1804d0032 -i td/osd-fast-mark-down/2/new.json 2023-11-02T09:08:37.460 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:08:37.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/2/new.json 2023-11-02T09:08:37.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=871ba12a-f05c-4576-8e0f-2a15553d0ce9 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.220526/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAUZ0NlxkqFORAA5oGoJ60zQN2GTTiWi7XMzw== --osd-uuid 58073f00-433f-4cc7-b61e-00c1804d0032 2023-11-02T09:08:37.501 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:37.497+0000 7f73fe727500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:08:37.502 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:37.499+0000 7f73fe727500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:08:37.503 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:37.500+0000 7f73fe727500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:08:37.504 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:37.501+0000 7f73fe727500 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2023-11-02T09:08:37.504 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:37.501+0000 7f73fe727500 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2023-11-02T09:08:37.505 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:37.501+0000 7f73fe727500 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2023-11-02T09:08:37.584 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:37.581+0000 7f73fe727500 -1 bluestore(td/osd-fast-mark-down/2) _read_fsid unparsable uuid 2023-11-02T09:08:43.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/2/keyring 2023-11-02T09:08:43.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:08:43.390 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2023-11-02T09:08:43.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-11-02T09:08:43.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:08:43.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-11-02T09:08:43.903 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T09:08:43.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=871ba12a-f05c-4576-8e0f-2a15553d0ce9 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.220526/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:08:43.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:08:43.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:08:43.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:08:43.929 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:43.926+0000 7fed0faaf500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:08:43.956 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:43.954+0000 7fed0faaf500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:08:43.957 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:43.955+0000 7fed0faaf500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:08:44.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-11-02T09:08:44.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:08:44.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T09:08:44.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:08:44.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:08:44.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:44.355 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:08:44.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:08:44.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:44.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:08:44.495 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:44.493+0000 7fed0faaf500 -1 Falling back to public interface 2023-11-02T09:08:44.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:45.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:45.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:45.764 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:08:45.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:08:45.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:45.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:08:46.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:46.843 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:08:46.840+0000 7fed0faaf500 -1 osd.2 0 log_to_monitors true 2023-11-02T09:08:47.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:47.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:47.191 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:08:47.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:08:47.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:47.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:08:47.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:48.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:48.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:48.618 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:08:48.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:08:48.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:48.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:08:49.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:50.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:50.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:50.027 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:08:50.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:08:50.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:50.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:08:50.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:51.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:51.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:51.412 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:08:51.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:08:51.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:51.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:08:51.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:52.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:52.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:52.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:08:52.877 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:08:52.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:52.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:08:53.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:54.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:54.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:54.300 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:08:54.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:08:54.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:54.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:08:54.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:55.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:55.684 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:08:55.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:55.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:08:55.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:55.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:08:56.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:57.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:57.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:57.182 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:08:57.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:08:57.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:57.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:08:57.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:08:58.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:08:58.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:08:58.582 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:08:58.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:08:58.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:08:58.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:08:59.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:00.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:00.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:00.328 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:09:00.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:09:00.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:00.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:09:01.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:02.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:02.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:02.432 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:09:02.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:09:02.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:02.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:09:02.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:03.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:03.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:03.929 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:09:03.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:09:03.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:03.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:09:04.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:05.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:05.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:05.450 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:09:05.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:09:05.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:05.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:09:06.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:07.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:07.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:07.160 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:09:07.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:09:07.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:07.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:09:07.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:08.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:08.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:08.680 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:09:08.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:09:08.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:08.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:09:09.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:10.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:10.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:10.357 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T09:09:10.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T09:09:10.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:10.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:09:10.750 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/225128,v1:127.0.0.1:6819/225128] [v2:127.0.0.1:6820/225128,v1:127.0.0.1:6821/225128] exists,up 58073f00-433f-4cc7-b61e-00c1804d0032 2023-11-02T09:09:10.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:09:10.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:09:10.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:09:10.752 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.2.pid 2023-11-02T09:09:10.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=225128 2023-11-02T09:09:10.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:56: test_fast_kill: create_rbd_pool 2023-11-02T09:09:10.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-11-02T09:09:11.122 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' does not exist 2023-11-02T09:09:11.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-11-02T09:09:11.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-11-02T09:09:11.794 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' already exists 2023-11-02T09:09:11.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:09:12.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-11-02T09:09:14.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:59: test_fast_kill: timeout 20 rados -p rbd bench 10 write -b 4096 --max-objects 128 --no-cleanup 2023-11-02T09:09:14.834 INFO:tasks.workunit.client.0.smithi116.stdout:hints = 1 2023-11-02T09:09:14.835 INFO:tasks.workunit.client.0.smithi116.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 128 objects 2023-11-02T09:09:14.835 INFO:tasks.workunit.client.0.smithi116.stdout:Object prefix: benchmark_data_smithi116_226098 2023-11-02T09:09:14.835 INFO:tasks.workunit.client.0.smithi116.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-11-02T09:09:14.835 INFO:tasks.workunit.client.0.smithi116.stdout: 0 0 0 0 0 0 - 0 2023-11-02T09:09:15.834 INFO:tasks.workunit.client.0.smithi116.stdout: 1 16 80 64 0.249974 0.25 0.236121 0.216503 2023-11-02T09:09:16.835 INFO:tasks.workunit.client.0.smithi116.stdout:Total time run: 1.84103 2023-11-02T09:09:16.835 INFO:tasks.workunit.client.0.smithi116.stdout:Total writes made: 128 2023-11-02T09:09:16.836 INFO:tasks.workunit.client.0.smithi116.stdout:Write size: 4096 2023-11-02T09:09:16.836 INFO:tasks.workunit.client.0.smithi116.stdout:Object size: 4096 2023-11-02T09:09:16.836 INFO:tasks.workunit.client.0.smithi116.stdout:Bandwidth (MB/sec): 0.271587 2023-11-02T09:09:16.837 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev Bandwidth: 0 2023-11-02T09:09:16.837 INFO:tasks.workunit.client.0.smithi116.stdout:Max bandwidth (MB/sec): 0.25 2023-11-02T09:09:16.837 INFO:tasks.workunit.client.0.smithi116.stdout:Min bandwidth (MB/sec): 0.25 2023-11-02T09:09:16.838 INFO:tasks.workunit.client.0.smithi116.stdout:Average IOPS: 69 2023-11-02T09:09:16.838 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev IOPS: 0 2023-11-02T09:09:16.838 INFO:tasks.workunit.client.0.smithi116.stdout:Max IOPS: 64 2023-11-02T09:09:16.839 INFO:tasks.workunit.client.0.smithi116.stdout:Min IOPS: 64 2023-11-02T09:09:16.839 INFO:tasks.workunit.client.0.smithi116.stdout:Average Latency(s): 0.228646 2023-11-02T09:09:16.839 INFO:tasks.workunit.client.0.smithi116.stdout:Stddev Latency(s): 0.0214798 2023-11-02T09:09:16.840 INFO:tasks.workunit.client.0.smithi116.stdout:Max latency(s): 0.259976 2023-11-02T09:09:16.840 INFO:tasks.workunit.client.0.smithi116.stdout:Min latency(s): 0.117563 2023-11-02T09:09:16.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:60: test_fast_kill: sleep 1 2023-11-02T09:09:17.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:62: test_fast_kill: killid=0 2023-11-02T09:09:17.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:63: test_fast_kill: previd=0 2023-11-02T09:09:17.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2023-11-02T09:09:17.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 0 -eq 0 ']' 2023-11-02T09:09:17.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=225128 2023-11-02T09:09:17.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 225128 -eq 0 ']' 2023-11-02T09:09:17.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=225128 2023-11-02T09:09:17.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 225128 2023-11-02T09:09:17.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2023-11-02T09:09:17.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2023-11-02T09:09:17.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2023-11-02T09:09:17.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2023-11-02T09:09:18.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2023-11-02T09:09:18.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2023-11-02T09:09:18.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2023-11-02T09:09:18.972 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2023-11-02T09:09:18.972 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2023-11-02T09:09:19.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=1 2023-11-02T09:09:19.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 1 -lt 1 ']' 2023-11-02T09:09:19.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 1 -gt 1 ']' 2023-11-02T09:09:19.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2023-11-02T09:09:19.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 1 -lt 1 ']' 2023-11-02T09:09:19.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2023-11-02T09:09:19.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 225128 -eq 225128 ']' 2023-11-02T09:09:19.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=221675 2023-11-02T09:09:19.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 221675 -eq 225128 ']' 2023-11-02T09:09:19.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=221675 2023-11-02T09:09:19.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 221675 2023-11-02T09:09:19.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2023-11-02T09:09:19.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2023-11-02T09:09:19.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2023-11-02T09:09:19.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2023-11-02T09:09:20.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2023-11-02T09:09:20.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2023-11-02T09:09:20.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2023-11-02T09:09:20.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2023-11-02T09:09:20.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2023-11-02T09:09:20.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=2 2023-11-02T09:09:20.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 2 -lt 2 ']' 2023-11-02T09:09:20.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 2 -gt 2 ']' 2023-11-02T09:09:20.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2023-11-02T09:09:20.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 2 -lt 2 ']' 2023-11-02T09:09:20.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:103: test_fast_kill: pkill -SIGTERM rados 2023-11-02T09:09:20.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:104: test_fast_kill: teardown td/osd-fast-mark-down 2023-11-02T09:09:20.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2023-11-02T09:09:20.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:09:20.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2023-11-02T09:09:20.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:09:20.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:09:20.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:09:20.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:09:20.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:09:20.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:09:20.986 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:09:20.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:09:20.987 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:09:20.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:09:20.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:09:20.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:09:20.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:09:20.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:09:20.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:09:20.990 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:09:20.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:09:20.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:09:20.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2023-11-02T09:09:21.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:09:21.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:09:21.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.220526 2023-11-02T09:09:21.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.220526 2023-11-02T09:09:21.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:09:21.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:09:21.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:42: run: return 0 2023-11-02T09:09:21.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2023-11-02T09:09:21.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-fast-mark-down 0 2023-11-02T09:09:21.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2023-11-02T09:09:21.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-11-02T09:09:21.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2023-11-02T09:09:21.009 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:09:21.009 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:09:21.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:09:21.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:09:21.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:09:21.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:09:21.019 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:09:21.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:09:21.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:09:21.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:09:21.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:09:21.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:09:21.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:09:21.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:09:21.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:09:21.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:09:21.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:09:21.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-11-02T09:09:21.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2023-11-02T09:09:21.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:09:21.026 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:09:21.026 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.220526 2023-11-02T09:09:21.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.220526 2023-11-02T09:09:21.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:09:21.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:09:21.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2023-11-02T09:09:21.027 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-02T09:09:21.027 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-02T09:09:21.088 INFO:tasks.workunit:Running workunit osd/osd-force-create-pg.sh... 2023-11-02T09:09:21.088 DEBUG:teuthology.orchestra.run.smithi116:workunit test osd/osd-force-create-pg.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh 2023-11-02T09:09:21.148 INFO:tasks.workunit.client.0.smithi116.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-11-02T09:09:21.154 INFO:tasks.workunit.client.0.smithi116.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-11-02T09:09:21.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T09:09:21.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T09:09:21.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2023-11-02T09:09:21.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2023-11-02T09:09:21.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2023-11-02T09:09:21.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2023-11-02T09:09:21.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2023-11-02T09:09:21.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2023-11-02T09:09:21.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-force-create-pg 2023-11-02T09:09:21.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:5: run: local dir=td/osd-force-create-pg 2023-11-02T09:09:21.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:6: run: shift 2023-11-02T09:09:21.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: export CEPH_MON=127.0.0.1:7145 2023-11-02T09:09:21.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: CEPH_MON=127.0.0.1:7145 2023-11-02T09:09:21.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:9: run: export CEPH_ARGS 2023-11-02T09:09:21.158 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: uuidgen 2023-11-02T09:09:21.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: CEPH_ARGS+='--fsid=5f8656fa-ecda-4bba-8de2-b9e334ec2a80 --auth-supported=none ' 2023-11-02T09:09:21.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7145 ' 2023-11-02T09:09:21.158 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: set 2023-11-02T09:09:21.158 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-11-02T09:09:21.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: local funcs=TEST_reuse_id 2023-11-02T09:09:21.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:14: run: for func in $funcs 2023-11-02T09:09:21.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:15: run: setup td/osd-force-create-pg 2023-11-02T09:09:21.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-force-create-pg 2023-11-02T09:09:21.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-force-create-pg 2023-11-02T09:09:21.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2023-11-02T09:09:21.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:09:21.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2023-11-02T09:09:21.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:09:21.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:09:21.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:09:21.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:09:21.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:09:21.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:09:21.166 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:09:21.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:09:21.166 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:09:21.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:09:21.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:09:21.167 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:09:21.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:09:21.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:09:21.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:09:21.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:09:21.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:09:21.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:09:21.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2023-11-02T09:09:21.172 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:09:21.172 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:09:21.173 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.226260 2023-11-02T09:09:21.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.226260 2023-11-02T09:09:21.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:09:21.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:09:21.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-force-create-pg 2023-11-02T09:09:21.174 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T09:09:21.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:09:21.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.226260 2023-11-02T09:09:21.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.226260 2023-11-02T09:09:21.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T09:09:21.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-11-02T09:09:21.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-11-02T09:09:21.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T09:09:21.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-force-create-pg 1' TERM HUP INT 2023-11-02T09:09:21.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:16: run: TEST_reuse_id td/osd-force-create-pg 2023-11-02T09:09:21.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:22: TEST_reuse_id: local dir=td/osd-force-create-pg 2023-11-02T09:09:21.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:24: TEST_reuse_id: run_mon td/osd-force-create-pg a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-11-02T09:09:21.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-force-create-pg 2023-11-02T09:09:21.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T09:09:21.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T09:09:21.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T09:09:21.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-force-create-pg/a 2023-11-02T09:09:21.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-force-create-pg/a --run-dir=td/osd-force-create-pg --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-11-02T09:09:21.545 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T09:09:21.546 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:09:21.546 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:09:21.546 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:09:21.546 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:09:21.547 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.226260 2023-11-02T09:09:21.547 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.226260/$cluster-$name.asok' 2023-11-02T09:09:21.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-force-create-pg/a '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.226260/$cluster-$name.asok' --mon-cluster-log-file=td/osd-force-create-pg/log --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-11-02T09:09:21.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T09:09:21.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T09:09:21.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:09:21.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:09:21.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T09:09:21.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T09:09:21.715 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:09:21.716 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:09:21.716 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:09:21.716 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:09:21.716 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:09:21.716 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.226260 2023-11-02T09:09:21.717 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.226260/ceph-mon.a.asok 2023-11-02T09:09:21.717 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:09:21.717 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.226260/ceph-mon.a.asok config get fsid 2023-11-02T09:09:21.797 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T09:09:21.797 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:09:21.797 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:09:21.798 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T09:09:21.798 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T09:09:21.798 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:09:21.798 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:09:21.798 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:09:21.799 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:09:21.799 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:09:21.799 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.226260 2023-11-02T09:09:21.799 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.226260/ceph-mon.a.asok 2023-11-02T09:09:21.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:09:21.800 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.226260/ceph-mon.a.asok config get mon_host 2023-11-02T09:09:21.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:25: TEST_reuse_id: run_mgr td/osd-force-create-pg x 2023-11-02T09:09:21.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-force-create-pg 2023-11-02T09:09:21.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T09:09:21.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T09:09:21.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T09:09:21.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-force-create-pg/x 2023-11-02T09:09:21.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T09:09:22.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T09:09:22.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:09:22.720 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:09:22.720 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:09:22.720 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:09:22.720 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.226260 2023-11-02T09:09:22.720 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.226260/$cluster-$name.asok' 2023-11-02T09:09:22.721 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:09:22.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-force-create-pg/x '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.226260/$cluster-$name.asok' --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:09:22.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:26: TEST_reuse_id: run_osd td/osd-force-create-pg 0 2023-11-02T09:09:22.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2023-11-02T09:09:22.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:09:22.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T09:09:22.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:09:22.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/0 2023-11-02T09:09:22.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5f8656fa-ecda-4bba-8de2-b9e334ec2a80 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2023-11-02T09:09:22.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:09:22.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:09:22.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:09:22.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2023-11-02T09:09:22.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2023-11-02T09:09:22.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:09:22.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:09:22.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2023-11-02T09:09:22.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:09:22.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:09:22.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:09:22.755 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:09:22.755 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:09:22.755 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.226260 2023-11-02T09:09:22.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.226260/$cluster-$name.asok' 2023-11-02T09:09:22.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.226260/$cluster-$name.asok' 2023-11-02T09:09:22.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:09:22.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:09:22.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:09:22.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2023-11-02T09:09:22.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2023-11-02T09:09:22.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:09:22.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:09:22.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:09:22.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:09:22.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:09:22.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:09:22.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/0 2023-11-02T09:09:22.758 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:09:22.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2c765320-afd1-4efb-8773-f040c8076522 2023-11-02T09:09:22.759 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 2c765320-afd1-4efb-8773-f040c8076522 2023-11-02T09:09:22.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2c765320-afd1-4efb-8773-f040c8076522' 2023-11-02T09:09:22.760 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:09:22.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBCZ0NlC4pSLRAAjM23OAm4uNS2LkrJkrpx9A== 2023-11-02T09:09:22.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBCZ0NlC4pSLRAAjM23OAm4uNS2LkrJkrpx9A=="}' 2023-11-02T09:09:22.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2c765320-afd1-4efb-8773-f040c8076522 -i td/osd-force-create-pg/0/new.json 2023-11-02T09:09:23.105 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:09:23.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/0/new.json 2023-11-02T09:09:23.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5f8656fa-ecda-4bba-8de2-b9e334ec2a80 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.226260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBCZ0NlC4pSLRAAjM23OAm4uNS2LkrJkrpx9A== --osd-uuid 2c765320-afd1-4efb-8773-f040c8076522 2023-11-02T09:09:23.142 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:09:23.138+0000 7f61db2c7500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:09:23.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:09:23.140+0000 7f61db2c7500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:09:23.144 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:09:23.141+0000 7f61db2c7500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:09:23.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:09:23.141+0000 7f61db2c7500 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2023-11-02T09:09:23.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:09:23.141+0000 7f61db2c7500 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2023-11-02T09:09:23.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:09:23.141+0000 7f61db2c7500 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2023-11-02T09:09:23.179 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:09:23.175+0000 7f61db2c7500 -1 bluestore(td/osd-force-create-pg/0) _read_fsid unparsable uuid 2023-11-02T09:09:28.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/0/keyring 2023-11-02T09:09:28.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:09:28.423 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T09:09:28.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T09:09:28.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:09:28.958 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T09:09:28.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T09:09:28.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5f8656fa-ecda-4bba-8de2-b9e334ec2a80 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.226260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:09:28.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:09:28.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:09:28.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:09:28.982 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:09:28.979+0000 7f88ad1e2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:09:28.997 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:09:28.992+0000 7f88ad1e2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:09:28.997 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:09:28.993+0000 7f88ad1e2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:09:29.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T09:09:29.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:09:29.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T09:09:29.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:09:29.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:09:29.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:29.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:09:29.359 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:09:29.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:29.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:09:29.517 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:09:29.513+0000 7f88ad1e2500 -1 Falling back to public interface 2023-11-02T09:09:29.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:30.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:30.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:30.740 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:09:30.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:09:30.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:30.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:09:31.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:31.829 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:09:31.826+0000 7f88ad1e2500 -1 osd.0 0 log_to_monitors true 2023-11-02T09:09:32.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:32.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:32.160 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:09:32.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:09:32.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:32.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:09:32.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:33.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:33.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:33.650 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:09:33.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:09:33.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:33.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:09:34.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:35.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:35.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:35.050 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:09:35.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:09:35.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:35.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:09:35.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:36.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:36.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:36.428 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:09:36.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:09:36.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:36.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:09:36.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:37.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:37.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:37.990 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:09:37.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:09:37.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:37.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:09:38.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:39.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:39.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:39.387 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:09:39.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:09:39.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:39.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:09:39.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:40.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:40.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:40.903 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:09:40.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:09:40.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:40.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:09:41.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:42.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:42.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:42.301 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:09:42.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:09:42.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:42.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:09:42.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:43.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:43.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:43.712 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:09:43.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:09:43.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:43.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:09:44.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:45.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:45.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:45.482 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:09:45.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:09:45.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:45.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:09:45.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:46.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:46.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:46.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:09:46.940 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:09:46.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:46.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:09:47.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:48.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:48.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:48.859 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:09:48.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:09:48.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:48.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:09:49.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:50.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:50.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:50.790 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:09:50.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:09:50.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:50.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:09:51.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:52.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:52.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:52.735 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:09:52.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:09:52.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:52.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:09:53.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:09:54.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:09:54.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:09:54.330 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:09:54.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:09:54.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:09:54.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:09:54.767 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/227344,v1:127.0.0.1:6803/227344] [v2:127.0.0.1:6804/227344,v1:127.0.0.1:6805/227344] exists,up 2c765320-afd1-4efb-8773-f040c8076522 2023-11-02T09:09:54.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:09:54.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:09:54.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:09:54.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:27: TEST_reuse_id: run_osd td/osd-force-create-pg 1 2023-11-02T09:09:54.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2023-11-02T09:09:54.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:09:54.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-11-02T09:09:54.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:09:54.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/1 2023-11-02T09:09:54.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5f8656fa-ecda-4bba-8de2-b9e334ec2a80 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2023-11-02T09:09:54.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:09:54.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:09:54.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:09:54.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2023-11-02T09:09:54.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2023-11-02T09:09:54.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:09:54.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:09:54.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2023-11-02T09:09:54.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:09:54.776 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:09:54.776 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:09:54.776 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:09:54.776 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:09:54.776 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.226260 2023-11-02T09:09:54.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.226260/$cluster-$name.asok' 2023-11-02T09:09:54.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.226260/$cluster-$name.asok' 2023-11-02T09:09:54.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:09:54.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:09:54.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:09:54.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2023-11-02T09:09:54.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2023-11-02T09:09:54.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:09:54.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:09:54.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:09:54.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:09:54.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:09:54.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:09:54.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/1 2023-11-02T09:09:54.779 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:09:54.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f14153c5-cefc-44db-9462-ae8bfc9f182a 2023-11-02T09:09:54.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f14153c5-cefc-44db-9462-ae8bfc9f182a' 2023-11-02T09:09:54.780 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 f14153c5-cefc-44db-9462-ae8bfc9f182a 2023-11-02T09:09:54.781 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:09:54.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBiZ0Nl4aidLhAA5mTykVeGFowrX5Y1GpxOLw== 2023-11-02T09:09:54.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBiZ0Nl4aidLhAA5mTykVeGFowrX5Y1GpxOLw=="}' 2023-11-02T09:09:54.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f14153c5-cefc-44db-9462-ae8bfc9f182a -i td/osd-force-create-pg/1/new.json 2023-11-02T09:09:55.308 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:09:55.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/1/new.json 2023-11-02T09:09:55.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5f8656fa-ecda-4bba-8de2-b9e334ec2a80 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.226260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBiZ0Nl4aidLhAA5mTykVeGFowrX5Y1GpxOLw== --osd-uuid f14153c5-cefc-44db-9462-ae8bfc9f182a 2023-11-02T09:09:55.345 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:09:55.342+0000 7fbc76b07500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:09:55.363 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:09:55.344+0000 7fbc76b07500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:09:55.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:09:55.345+0000 7fbc76b07500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:09:55.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:09:55.345+0000 7fbc76b07500 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2023-11-02T09:09:55.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:09:55.345+0000 7fbc76b07500 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2023-11-02T09:09:55.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:09:55.345+0000 7fbc76b07500 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2023-11-02T09:09:55.399 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:09:55.396+0000 7fbc76b07500 -1 bluestore(td/osd-force-create-pg/1) _read_fsid unparsable uuid 2023-11-02T09:10:00.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/1/keyring 2023-11-02T09:10:00.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:10:00.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-11-02T09:10:00.778 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2023-11-02T09:10:00.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:10:01.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-11-02T09:10:01.309 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T09:10:01.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5f8656fa-ecda-4bba-8de2-b9e334ec2a80 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.226260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:10:01.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:10:01.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:10:01.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:10:01.332 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:10:01.330+0000 7f2c68c10500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:10:01.355 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:10:01.352+0000 7f2c68c10500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:10:01.356 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:10:01.353+0000 7f2c68c10500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:10:01.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-11-02T09:10:01.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:10:01.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T09:10:01.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:10:01.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:10:01.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:01.753 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:10:01.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:10:01.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:01.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:10:01.888 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:10:01.885+0000 7f2c68c10500 -1 Falling back to public interface 2023-11-02T09:10:02.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:03.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:03.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:03.154 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:10:03.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:10:03.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:03.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:10:03.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:04.140 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:10:04.137+0000 7f2c68c10500 -1 osd.1 0 log_to_monitors true 2023-11-02T09:10:04.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:04.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:04.536 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:10:04.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:10:04.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:04.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:10:05.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:06.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:06.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:06.036 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:10:06.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:10:06.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:06.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:10:06.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:07.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:07.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:07.474 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:10:07.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:10:07.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:07.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:10:07.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:08.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:08.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:08.965 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:10:08.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:10:08.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:08.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:10:09.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:10.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:10.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:10.365 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:10:10.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:10:10.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:10.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:10:10.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:11.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:11.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:11.743 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:10:11.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:10:11.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:11.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:10:12.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:13.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:13.184 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:10:13.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:13.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:10:13.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:13.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:10:13.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:14.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:14.579 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:10:14.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:14.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:10:14.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:14.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:10:15.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:16.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:16.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:16.077 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:10:16.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:10:16.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:16.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:10:16.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:17.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:17.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:17.661 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:10:17.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:10:17.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:17.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:10:18.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:19.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:19.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:19.592 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:10:19.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:10:19.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:19.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:10:20.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:21.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:21.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:21.388 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:10:21.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:10:21.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:21.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:10:22.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:23.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:23.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:23.443 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:10:23.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:10:23.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:23.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:10:24.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:25.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:25.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:25.469 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:10:25.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:10:25.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:25.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:10:26.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:27.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:27.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:27.275 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:10:27.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:10:27.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:27.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:10:27.688 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/229057,v1:127.0.0.1:6811/229057] [v2:127.0.0.1:6812/229057,v1:127.0.0.1:6813/229057] exists,up f14153c5-cefc-44db-9462-ae8bfc9f182a 2023-11-02T09:10:27.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:10:27.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:10:27.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:10:27.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:28: TEST_reuse_id: run_osd td/osd-force-create-pg 2 2023-11-02T09:10:27.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2023-11-02T09:10:27.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:10:27.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-11-02T09:10:27.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:10:27.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/2 2023-11-02T09:10:27.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5f8656fa-ecda-4bba-8de2-b9e334ec2a80 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2023-11-02T09:10:27.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:10:27.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:10:27.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:10:27.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2023-11-02T09:10:27.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2023-11-02T09:10:27.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:10:27.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:10:27.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2023-11-02T09:10:27.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:10:27.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:10:27.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:10:27.709 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:10:27.709 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:10:27.710 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.226260 2023-11-02T09:10:27.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.226260/$cluster-$name.asok' 2023-11-02T09:10:27.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.226260/$cluster-$name.asok' 2023-11-02T09:10:27.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:10:27.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:10:27.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:10:27.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2023-11-02T09:10:27.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2023-11-02T09:10:27.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:10:27.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:10:27.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:10:27.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:10:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:10:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:10:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/2 2023-11-02T09:10:27.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:10:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4f8f81fb-14f0-4887-a7cb-80a4e162c477 2023-11-02T09:10:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4f8f81fb-14f0-4887-a7cb-80a4e162c477' 2023-11-02T09:10:27.717 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 4f8f81fb-14f0-4887-a7cb-80a4e162c477 2023-11-02T09:10:27.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:10:27.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCDZ0NllxANKhAAxWp4uD4FbShp/IzOpwfoKA== 2023-11-02T09:10:27.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCDZ0NllxANKhAAxWp4uD4FbShp/IzOpwfoKA=="}' 2023-11-02T09:10:27.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4f8f81fb-14f0-4887-a7cb-80a4e162c477 -i td/osd-force-create-pg/2/new.json 2023-11-02T09:10:28.208 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:10:28.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/2/new.json 2023-11-02T09:10:28.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5f8656fa-ecda-4bba-8de2-b9e334ec2a80 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.226260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCDZ0NllxANKhAAxWp4uD4FbShp/IzOpwfoKA== --osd-uuid 4f8f81fb-14f0-4887-a7cb-80a4e162c477 2023-11-02T09:10:28.246 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:10:28.243+0000 7f701740c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:10:28.247 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:10:28.244+0000 7f701740c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:10:28.248 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:10:28.245+0000 7f701740c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:10:28.249 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:10:28.246+0000 7f701740c500 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2023-11-02T09:10:28.249 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:10:28.246+0000 7f701740c500 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2023-11-02T09:10:28.250 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:10:28.246+0000 7f701740c500 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2023-11-02T09:10:28.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:10:28.312+0000 7f701740c500 -1 bluestore(td/osd-force-create-pg/2) _read_fsid unparsable uuid 2023-11-02T09:10:34.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/2/keyring 2023-11-02T09:10:34.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:10:34.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-11-02T09:10:34.151 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2023-11-02T09:10:34.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:10:34.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-11-02T09:10:34.661 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T09:10:34.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5f8656fa-ecda-4bba-8de2-b9e334ec2a80 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.226260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:10:34.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:10:34.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:10:34.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:10:34.684 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:10:34.681+0000 7f1d38a0c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:10:34.713 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:10:34.710+0000 7f1d38a0c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:10:34.714 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:10:34.711+0000 7f1d38a0c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:10:35.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-11-02T09:10:35.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:10:35.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T09:10:35.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:10:35.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:10:35.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:35.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:10:35.031 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:10:35.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:35.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:10:35.253 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:10:35.251+0000 7f1d38a0c500 -1 Falling back to public interface 2023-11-02T09:10:35.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:36.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:36.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:36.470 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:10:36.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:10:36.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:36.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:10:36.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:37.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:10:37.504+0000 7f1d38a0c500 -1 osd.2 0 log_to_monitors true 2023-11-02T09:10:37.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:37.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:37.856 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:10:37.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:10:37.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:37.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:10:38.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:39.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:39.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:39.370 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:10:39.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:10:39.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:39.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:10:39.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:40.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:40.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:40.782 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:10:40.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:10:40.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:40.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:10:41.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:42.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:42.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:42.239 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:10:42.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:10:42.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:42.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:10:42.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:43.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:43.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:43.618 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:10:43.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:10:43.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:43.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:10:43.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:44.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:44.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:44.998 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:10:44.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:10:44.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:44.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:10:45.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:46.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:46.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:46.525 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:10:46.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:10:46.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:46.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:10:46.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:47.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:47.905 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:10:47.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:47.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:10:47.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:47.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:10:48.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:49.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:49.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:49.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:10:49.450 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:10:49.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:49.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:10:49.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:50.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:50.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:50.893 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:10:50.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:10:50.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:50.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:10:51.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:52.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:52.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:52.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:10:52.422 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:10:52.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:52.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:10:52.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:53.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:53.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:53.920 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:10:53.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:10:53.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:53.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:10:54.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:55.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:55.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:55.771 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:10:55.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:10:55.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:55.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:10:56.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:57.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:57.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:57.484 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:10:57.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:10:57.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:57.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:10:58.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:10:59.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:10:59.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:10:59.209 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:10:59.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:10:59.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:10:59.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:11:00.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:11:01.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:11:01.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:11:01.103 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T09:11:01.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T09:11:01.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:11:01.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:11:01.504 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/230773,v1:127.0.0.1:6819/230773] [v2:127.0.0.1:6820/230773,v1:127.0.0.1:6821/230773] exists,up 4f8f81fb-14f0-4887-a7cb-80a4e162c477 2023-11-02T09:11:01.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:11:01.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:11:01.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:11:01.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:30: TEST_reuse_id: ceph osd pool create foo 50 2023-11-02T09:11:02.043 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'foo' already exists 2023-11-02T09:11:02.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:31: TEST_reuse_id: wait_for_clean 2023-11-02T09:11:02.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T09:11:02.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T09:11:02.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T09:11:02.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-11-02T09:11:02.057 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T09:11:02.057 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T09:11:02.058 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T09:11:02.058 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T09:11:02.058 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T09:11:02.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-11-02T09:11:02.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T09:11:02.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T09:11:02.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T09:11:02.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T09:11:02.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T09:11:02.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T09:11:02.603 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T09:11:02.603 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T09:11:02.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T09:11:02.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:11:02.604 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T09:11:02.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836501 2023-11-02T09:11:02.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836501 2023-11-02T09:11:02.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836501' 2023-11-02T09:11:02.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:11:02.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T09:11:03.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672975 2023-11-02T09:11:03.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672975 2023-11-02T09:11:03.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836501 1-42949672975' 2023-11-02T09:11:03.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:11:03.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T09:11:03.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509448 2023-11-02T09:11:03.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509448 2023-11-02T09:11:03.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836501 1-42949672975 2-64424509448' 2023-11-02T09:11:03.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:11:03.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836501 2023-11-02T09:11:03.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:11:03.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T09:11:03.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836501 2023-11-02T09:11:03.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:11:03.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836501 2023-11-02T09:11:03.204 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836501 2023-11-02T09:11:03.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2023-11-02T09:11:03.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:11:03.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836499 -lt 21474836501 2023-11-02T09:11:03.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T09:11:04.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T09:11:04.760 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:11:05.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836502 -lt 21474836501 2023-11-02T09:11:05.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:11:05.135 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672975 2023-11-02T09:11:05.135 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:11:05.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T09:11:05.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672975 2023-11-02T09:11:05.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:11:05.137 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672975 2023-11-02T09:11:05.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672975 2023-11-02T09:11:05.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2023-11-02T09:11:05.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T09:11:05.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672975 -lt 42949672975 2023-11-02T09:11:05.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:11:05.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509448 2023-11-02T09:11:05.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:11:05.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T09:11:05.503 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509448 2023-11-02T09:11:05.503 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:11:05.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509448 2023-11-02T09:11:05.504 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509448 2023-11-02T09:11:05.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2023-11-02T09:11:05.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T09:11:05.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509448 -lt 64424509448 2023-11-02T09:11:05.907 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T09:11:05.908 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:11:05.908 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:11:06.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 50 == 0 2023-11-02T09:11:06.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:11:06.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:11:06.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:11:06.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:11:06.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:11:06.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:11:06.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:11:06.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=50 2023-11-02T09:11:06.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:11:06.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:11:06.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:11:07.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 50 = 50 2023-11-02T09:11:07.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T09:11:07.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T09:11:07.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:33: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.0 2023-11-02T09:11:07.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:11:07.316 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:11:07.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:11:07.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:11:07.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:11:07.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:11:07.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:34: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.1 2023-11-02T09:11:07.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:11:07.622 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:11:07.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:11:07.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:11:07.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:11:07.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:11:07.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:35: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.2 2023-11-02T09:11:07.928 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:11:07.928 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:11:07.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:11:07.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:11:07.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:11:08.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:11:08.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:36: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/0 --op remove --pgid 1.0 --force 2023-11-02T09:11:09.353 INFO:tasks.workunit.client.0.smithi116.stderr:PG '1.0' not found 2023-11-02T09:11:09.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:37: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/1 --op remove --pgid 1.0 --force 2023-11-02T09:11:10.635 INFO:tasks.workunit.client.0.smithi116.stdout: marking collection for removal 2023-11-02T09:11:10.635 INFO:tasks.workunit.client.0.smithi116.stdout:setting '_remove' omap key 2023-11-02T09:11:10.636 INFO:tasks.workunit.client.0.smithi116.stdout:finish_remove_pgs 1.0_head removing 1.0 2023-11-02T09:11:10.785 INFO:tasks.workunit.client.0.smithi116.stdout:Remove successful 2023-11-02T09:11:11.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:38: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/2 --op remove --pgid 1.0 --force 2023-11-02T09:11:12.139 INFO:tasks.workunit.client.0.smithi116.stderr:PG '1.0' not found 2023-11-02T09:11:12.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:39: TEST_reuse_id: activate_osd td/osd-force-create-pg 0 2023-11-02T09:11:12.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2023-11-02T09:11:12.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T09:11:12.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-11-02T09:11:12.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T09:11:12.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/0 2023-11-02T09:11:12.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5f8656fa-ecda-4bba-8de2-b9e334ec2a80 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2023-11-02T09:11:12.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:11:12.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:11:12.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:11:12.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2023-11-02T09:11:12.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2023-11-02T09:11:12.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T09:11:12.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T09:11:12.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2023-11-02T09:11:12.417 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T09:11:12.417 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:11:12.417 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:11:12.417 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:11:12.417 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:11:12.417 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.226260 2023-11-02T09:11:12.418 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.226260/$cluster-$name.asok' 2023-11-02T09:11:12.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.226260/$cluster-$name.asok' 2023-11-02T09:11:12.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:11:12.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2023-11-02T09:11:12.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2023-11-02T09:11:12.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:11:12.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:11:12.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:11:12.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:11:12.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T09:11:12.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T09:11:12.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/0 2023-11-02T09:11:12.421 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T09:11:12.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-11-02T09:11:12.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=5f8656fa-ecda-4bba-8de2-b9e334ec2a80 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.226260/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:11:12.422 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/0/whoami 2023-11-02T09:11:12.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-11-02T09:11:12.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T09:11:12.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T09:11:12.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T09:11:12.441 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:12.437+0000 7ff09bc0c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:11:12.442 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:12.439+0000 7ff09bc0c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:11:12.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:12.441+0000 7ff09bc0c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:11:12.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-11-02T09:11:12.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:11:12.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T09:11:12.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:11:12.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:11:12.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:11:12.812 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:11:12.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:11:12.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:11:12.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:11:12.960 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:12.957+0000 7ff09bc0c500 -1 Falling back to public interface 2023-11-02T09:11:13.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:11:14.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:11:14.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:11:14.210 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:11:14.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:11:14.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:11:14.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:11:14.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:11:15.354 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:15.352+0000 7ff09bc0c500 -1 osd.0 22 log_to_monitors true 2023-11-02T09:11:15.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:11:15.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:11:15.632 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:11:15.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:11:15.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:11:15.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:11:16.032 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 27 up_thru 27 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/233225,v1:127.0.0.1:6803/233225] [v2:127.0.0.1:6804/233225,v1:127.0.0.1:6805/233225] exists,up 2c765320-afd1-4efb-8773-f040c8076522 2023-11-02T09:11:16.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:11:16.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:11:16.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:11:16.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:40: TEST_reuse_id: activate_osd td/osd-force-create-pg 1 2023-11-02T09:11:16.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2023-11-02T09:11:16.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T09:11:16.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2023-11-02T09:11:16.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T09:11:16.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/1 2023-11-02T09:11:16.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5f8656fa-ecda-4bba-8de2-b9e334ec2a80 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2023-11-02T09:11:16.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:11:16.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:11:16.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:11:16.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2023-11-02T09:11:16.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2023-11-02T09:11:16.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T09:11:16.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T09:11:16.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2023-11-02T09:11:16.040 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T09:11:16.040 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:11:16.040 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:11:16.040 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:11:16.040 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:11:16.041 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.226260 2023-11-02T09:11:16.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.226260/$cluster-$name.asok' 2023-11-02T09:11:16.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.226260/$cluster-$name.asok' 2023-11-02T09:11:16.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:11:16.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2023-11-02T09:11:16.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2023-11-02T09:11:16.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:11:16.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:11:16.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:11:16.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:11:16.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T09:11:16.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T09:11:16.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/1 2023-11-02T09:11:16.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2023-11-02T09:11:16.044 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T09:11:16.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=5f8656fa-ecda-4bba-8de2-b9e334ec2a80 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.226260/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:11:16.045 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/1/whoami 2023-11-02T09:11:16.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2023-11-02T09:11:16.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T09:11:16.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T09:11:16.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T09:11:16.058 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:16.056+0000 7f0b79774500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:11:16.061 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:16.058+0000 7f0b79774500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:11:16.062 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:16.059+0000 7f0b79774500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:11:16.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2023-11-02T09:11:16.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:11:16.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T09:11:16.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:11:16.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:11:16.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:11:16.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:11:16.456 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:11:16.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:11:16.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:11:16.577 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:16.574+0000 7f0b79774500 -1 Falling back to public interface 2023-11-02T09:11:16.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:11:17.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:11:17.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:11:17.854 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:11:17.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:11:17.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:11:17.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:11:18.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:11:19.143 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:19.141+0000 7f0b79774500 -1 osd.1 23 log_to_monitors true 2023-11-02T09:11:19.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:11:19.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:11:19.254 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:11:19.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:11:19.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:11:19.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:11:19.640 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 30 up_thru 17 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6810/233769,v1:127.0.0.1:6811/233769] [v2:127.0.0.1:6812/233769,v1:127.0.0.1:6813/233769] exists,up f14153c5-cefc-44db-9462-ae8bfc9f182a 2023-11-02T09:11:19.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:11:19.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:11:19.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:11:19.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:41: TEST_reuse_id: activate_osd td/osd-force-create-pg 2 2023-11-02T09:11:19.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2023-11-02T09:11:19.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T09:11:19.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2023-11-02T09:11:19.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T09:11:19.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/2 2023-11-02T09:11:19.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5f8656fa-ecda-4bba-8de2-b9e334ec2a80 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2023-11-02T09:11:19.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:11:19.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:11:19.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:11:19.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2023-11-02T09:11:19.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2023-11-02T09:11:19.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T09:11:19.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T09:11:19.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2023-11-02T09:11:19.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T09:11:19.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:11:19.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:11:19.650 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:11:19.651 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:11:19.651 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.226260 2023-11-02T09:11:19.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.226260/$cluster-$name.asok' 2023-11-02T09:11:19.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.226260/$cluster-$name.asok' 2023-11-02T09:11:19.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:11:19.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2023-11-02T09:11:19.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2023-11-02T09:11:19.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:11:19.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:11:19.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:11:19.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:11:19.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T09:11:19.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T09:11:19.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/2 2023-11-02T09:11:19.654 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T09:11:19.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2023-11-02T09:11:19.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=5f8656fa-ecda-4bba-8de2-b9e334ec2a80 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.226260/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:11:19.656 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/2/whoami 2023-11-02T09:11:19.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2023-11-02T09:11:19.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T09:11:19.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T09:11:19.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T09:11:19.670 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:19.667+0000 7f7d4c21a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:11:19.671 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:19.669+0000 7f7d4c21a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:11:19.673 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:19.670+0000 7f7d4c21a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:11:20.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2023-11-02T09:11:20.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:11:20.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T09:11:20.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:11:20.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:11:20.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:11:20.153 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:11:20.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:11:20.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:11:20.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:11:20.208 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:20.204+0000 7f7d4c21a500 -1 Falling back to public interface 2023-11-02T09:11:20.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:11:21.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:11:21.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:11:21.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:11:21.534 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:11:21.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:11:21.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:11:21.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:11:22.536 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:22.533+0000 7f7d4c21a500 -1 osd.2 24 log_to_monitors true 2023-11-02T09:11:22.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:11:22.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:11:22.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:11:22.944 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:11:22.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:11:22.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:11:23.435 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 33 up_thru 33 down_at 25 last_clean_interval [15,24) [v2:127.0.0.1:6818/234313,v1:127.0.0.1:6819/234313] [v2:127.0.0.1:6820/234313,v1:127.0.0.1:6821/234313] exists,up 4f8f81fb-14f0-4887-a7cb-80a4e162c477 2023-11-02T09:11:23.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:11:23.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:11:23.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:11:23.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: sleep 10 2023-11-02T09:11:33.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: ceph pg ls 2023-11-02T09:11:33.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: grep 1.0 2023-11-02T09:11:33.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: grep stale 2023-11-02T09:11:33.807 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 0 0 0 0 0 0 0 0 0 stale+active+clean 30s 0'0 21:12 [1]p1 [1]p1 2023-11-02T09:11:01.809362+0000 2023-11-02T09:11:01.809362+0000 0 periodic scrub scheduled @ 2023-11-03T15:42:37.283219+0000 2023-11-02T09:11:33.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:45: TEST_reuse_id: ceph osd force-create-pg 1.0 --yes-i-really-mean-it 2023-11-02T09:11:34.378 INFO:tasks.workunit.client.0.smithi116.stderr:pg 1.0 already creating 2023-11-02T09:11:34.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:46: TEST_reuse_id: wait_for_clean 2023-11-02T09:11:34.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T09:11:34.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T09:11:34.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T09:11:34.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-11-02T09:11:34.390 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T09:11:34.391 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T09:11:34.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T09:11:34.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T09:11:34.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T09:11:34.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-11-02T09:11:34.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T09:11:34.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T09:11:34.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T09:11:34.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T09:11:34.470 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T09:11:34.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T09:11:34.877 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T09:11:34.877 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2023-11-02T09:11:34.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T09:11:34.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:11:34.877 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T09:11:34.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=115964116998 2023-11-02T09:11:34.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 115964116998 2023-11-02T09:11:34.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-115964116998' 2023-11-02T09:11:34.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:11:34.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T09:11:35.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018886 2023-11-02T09:11:35.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018886 2023-11-02T09:11:35.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-115964116998 1-128849018886' 2023-11-02T09:11:35.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:11:35.097 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T09:11:35.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=141733920773 2023-11-02T09:11:35.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 141733920773 2023-11-02T09:11:35.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-115964116998 1-128849018886 2-141733920773' 2023-11-02T09:11:35.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:11:35.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-115964116998 2023-11-02T09:11:35.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:11:35.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T09:11:35.206 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-115964116998 2023-11-02T09:11:35.206 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:11:35.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=115964116998 2023-11-02T09:11:35.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 115964116998' 2023-11-02T09:11:35.207 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 115964116998 2023-11-02T09:11:35.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:11:35.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 115964116996 -lt 115964116998 2023-11-02T09:11:35.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T09:11:36.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T09:11:36.611 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:11:37.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 115964116999 -lt 115964116998 2023-11-02T09:11:37.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:11:37.010 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-128849018886 2023-11-02T09:11:37.010 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:11:37.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T09:11:37.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-128849018886 2023-11-02T09:11:37.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:11:37.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018886 2023-11-02T09:11:37.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 128849018886' 2023-11-02T09:11:37.014 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 128849018886 2023-11-02T09:11:37.014 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T09:11:37.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018886 -lt 128849018886 2023-11-02T09:11:37.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:11:37.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-141733920773 2023-11-02T09:11:37.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:11:37.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T09:11:37.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-141733920773 2023-11-02T09:11:37.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:11:37.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=141733920773 2023-11-02T09:11:37.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 141733920773' 2023-11-02T09:11:37.391 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 141733920773 2023-11-02T09:11:37.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T09:11:37.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 141733920773 -lt 141733920773 2023-11-02T09:11:37.841 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T09:11:37.841 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:11:37.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:11:38.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 50 == 0 2023-11-02T09:11:38.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:11:38.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:11:38.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:11:38.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:11:38.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:11:38.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:11:38.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:11:38.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=50 2023-11-02T09:11:38.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:11:38.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:11:38.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:11:39.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 50 = 50 2023-11-02T09:11:39.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T09:11:39.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T09:11:39.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:17: run: teardown td/osd-force-create-pg 2023-11-02T09:11:39.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2023-11-02T09:11:39.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:11:39.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2023-11-02T09:11:39.207 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:11:39.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:11:39.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:11:39.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:11:39.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:11:39.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:11:39.313 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:11:39.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:11:39.314 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:11:39.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:11:39.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:11:39.317 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:11:39.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:11:39.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:11:39.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:11:39.320 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:11:39.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:11:39.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:11:39.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2023-11-02T09:11:39.333 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:11:39.334 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:11:39.334 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.226260 2023-11-02T09:11:39.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.226260 2023-11-02T09:11:39.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:11:39.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:11:39.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2023-11-02T09:11:39.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-force-create-pg 0 2023-11-02T09:11:39.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2023-11-02T09:11:39.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-11-02T09:11:39.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2023-11-02T09:11:39.339 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:11:39.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:11:39.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:11:39.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:11:39.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:11:39.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:11:39.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:11:39.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:11:39.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:11:39.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:11:39.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:11:39.346 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:11:39.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:11:39.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:11:39.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:11:39.347 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:11:39.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:11:39.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-11-02T09:11:39.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2023-11-02T09:11:39.349 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:11:39.349 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:11:39.349 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.226260 2023-11-02T09:11:39.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.226260 2023-11-02T09:11:39.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:11:39.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:11:39.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2023-11-02T09:11:39.351 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-02T09:11:39.351 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-02T09:11:39.416 INFO:tasks.workunit:Running workunit osd/osd-markdown.sh... 2023-11-02T09:11:39.417 DEBUG:teuthology.orchestra.run.smithi116:workunit test osd/osd-markdown.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh 2023-11-02T09:11:39.479 INFO:tasks.workunit.client.0.smithi116.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-11-02T09:11:39.485 INFO:tasks.workunit.client.0.smithi116.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-11-02T09:11:39.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T09:11:39.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T09:11:39.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2023-11-02T09:11:39.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2023-11-02T09:11:39.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2023-11-02T09:11:39.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2023-11-02T09:11:39.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2023-11-02T09:11:39.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2023-11-02T09:11:39.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-markdown 2023-11-02T09:11:39.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:22: run: local dir=td/osd-markdown 2023-11-02T09:11:39.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:23: run: shift 2023-11-02T09:11:39.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: export CEPH_MON=127.0.0.1:7108 2023-11-02T09:11:39.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: CEPH_MON=127.0.0.1:7108 2023-11-02T09:11:39.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:26: run: export CEPH_ARGS 2023-11-02T09:11:39.489 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: uuidgen 2023-11-02T09:11:39.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: CEPH_ARGS+='--fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none ' 2023-11-02T09:11:39.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7108 ' 2023-11-02T09:11:39.490 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: set 2023-11-02T09:11:39.490 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-11-02T09:11:39.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: local 'funcs=TEST_markdown_boot 2023-11-02T09:11:39.496 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_markdown_boot_exceed_time 2023-11-02T09:11:39.496 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_markdown_exceed_maxdown_count 2023-11-02T09:11:39.497 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_osd_stop' 2023-11-02T09:11:39.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2023-11-02T09:11:39.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2023-11-02T09:11:39.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2023-11-02T09:11:39.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2023-11-02T09:11:39.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2023-11-02T09:11:39.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:11:39.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2023-11-02T09:11:39.499 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:11:39.500 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:11:39.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:11:39.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:11:39.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:11:39.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:11:39.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:11:39.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:11:39.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:11:39.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:11:39.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:11:39.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:11:39.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:11:39.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:11:39.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:11:39.503 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:11:39.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:11:39.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:11:39.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2023-11-02T09:11:39.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:11:39.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:11:39.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:11:39.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.235308 2023-11-02T09:11:39.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:11:39.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:11:39.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2023-11-02T09:11:39.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T09:11:39.507 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:11:39.507 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:11:39.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.235308 2023-11-02T09:11:39.510 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T09:11:39.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-11-02T09:11:39.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-11-02T09:11:39.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T09:11:39.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2023-11-02T09:11:39.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot td/osd-markdown 2023-11-02T09:11:39.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:82: TEST_markdown_boot: local dir=td/osd-markdown 2023-11-02T09:11:39.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:84: TEST_markdown_boot: run_mon td/osd-markdown a 2023-11-02T09:11:39.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2023-11-02T09:11:39.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T09:11:39.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T09:11:39.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T09:11:39.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2023-11-02T09:11:39.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2023-11-02T09:11:39.847 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T09:11:39.847 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:11:39.848 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:11:39.848 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:11:39.848 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:11:39.848 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:11:39.848 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:11:39.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T09:11:40.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T09:11:40.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T09:11:40.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:11:40.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:11:40.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T09:11:40.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T09:11:40.017 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:11:40.017 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:11:40.017 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:11:40.018 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:11:40.018 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:11:40.018 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:11:40.018 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.235308/ceph-mon.a.asok 2023-11-02T09:11:40.018 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:11:40.019 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.235308/ceph-mon.a.asok config get fsid 2023-11-02T09:11:40.095 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T09:11:40.095 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:11:40.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:11:40.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T09:11:40.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T09:11:40.096 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:11:40.097 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:11:40.097 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:11:40.097 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:11:40.097 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:11:40.097 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:11:40.098 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.235308/ceph-mon.a.asok 2023-11-02T09:11:40.098 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:11:40.098 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.235308/ceph-mon.a.asok config get mon_host 2023-11-02T09:11:40.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:85: TEST_markdown_boot: run_mgr td/osd-markdown x 2023-11-02T09:11:40.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2023-11-02T09:11:40.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T09:11:40.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T09:11:40.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T09:11:40.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2023-11-02T09:11:40.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T09:11:41.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T09:11:41.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:11:41.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:11:41.012 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:11:41.012 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:11:41.012 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:11:41.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:11:41.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:11:41.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:11:41.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:86: TEST_markdown_boot: run_osd td/osd-markdown 0 2023-11-02T09:11:41.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2023-11-02T09:11:41.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:11:41.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T09:11:41.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:11:41.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2023-11-02T09:11:41.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 ' 2023-11-02T09:11:41.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:11:41.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:11:41.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:11:41.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2023-11-02T09:11:41.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2023-11-02T09:11:41.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:11:41.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:11:41.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2023-11-02T09:11:41.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:11:41.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:11:41.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:11:41.044 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:11:41.044 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:11:41.044 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:11:41.045 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:11:41.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:11:41.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:11:41.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:11:41.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:11:41.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2023-11-02T09:11:41.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2023-11-02T09:11:41.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:11:41.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:11:41.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:11:41.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:11:41.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:11:41.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:11:41.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2023-11-02T09:11:41.047 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:11:41.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9676b58c-01f5-4bd2-afc0-67039d857939 2023-11-02T09:11:41.048 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 9676b58c-01f5-4bd2-afc0-67039d857939 2023-11-02T09:11:41.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9676b58c-01f5-4bd2-afc0-67039d857939' 2023-11-02T09:11:41.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:11:41.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDNZ0Nl/xAgAxAAIGjov8k/c4jSSqAvDveWNw== 2023-11-02T09:11:41.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDNZ0Nl/xAgAxAAIGjov8k/c4jSSqAvDveWNw=="}' 2023-11-02T09:11:41.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9676b58c-01f5-4bd2-afc0-67039d857939 -i td/osd-markdown/0/new.json 2023-11-02T09:11:41.407 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:11:41.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2023-11-02T09:11:41.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDNZ0Nl/xAgAxAAIGjov8k/c4jSSqAvDveWNw== --osd-uuid 9676b58c-01f5-4bd2-afc0-67039d857939 2023-11-02T09:11:41.443 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:41.440+0000 7fbafda12500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:11:41.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:41.442+0000 7fbafda12500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:11:41.446 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:41.443+0000 7fbafda12500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:11:41.446 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:41.443+0000 7fbafda12500 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2023-11-02T09:11:41.447 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:41.444+0000 7fbafda12500 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2023-11-02T09:11:41.447 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:41.444+0000 7fbafda12500 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2023-11-02T09:11:41.498 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:41.495+0000 7fbafda12500 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2023-11-02T09:11:46.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2023-11-02T09:11:46.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:11:46.719 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T09:11:46.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T09:11:46.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:11:47.229 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T09:11:47.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T09:11:47.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:11:47.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:11:47.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:11:47.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:11:47.253 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:47.250+0000 7f0bbc939500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:11:47.278 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:47.275+0000 7f0bbc939500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:11:47.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:47.276+0000 7f0bbc939500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:11:47.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T09:11:47.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:11:47.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T09:11:47.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:11:47.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:11:47.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:11:47.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:11:47.639 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:11:47.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:11:47.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:11:47.812 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:47.808+0000 7f0bbc939500 -1 Falling back to public interface 2023-11-02T09:11:48.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:11:49.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:11:49.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:11:49.043 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:11:49.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:11:49.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:11:49.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:11:49.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:11:50.096 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:11:50.093+0000 7f0bbc939500 -1 osd.0 0 log_to_monitors true 2023-11-02T09:11:50.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:11:50.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:11:50.445 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:11:50.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:11:50.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:11:50.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:11:50.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:11:51.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:11:51.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:11:51.831 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:11:51.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:11:51.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:11:51.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:11:52.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:11:53.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:11:53.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:11:53.230 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:11:53.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:11:53.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:11:53.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:11:53.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:11:54.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:11:54.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:11:54.608 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:11:54.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:11:54.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:11:54.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:11:55.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:11:56.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:11:56.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:11:56.088 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:11:56.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:11:56.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:11:56.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:11:56.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:11:57.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:11:57.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:11:57.507 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:11:57.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:11:57.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:11:57.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:11:57.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:11:58.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:11:58.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:11:58.938 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:11:58.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:11:58.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:11:58.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:11:59.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:00.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:00.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:00.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:12:00.349 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:12:00.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:00.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:12:00.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:01.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:01.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:01.817 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:12:01.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:12:01.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:01.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:12:02.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:03.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:03.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:03.333 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:12:03.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:12:03.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:03.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:12:03.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:04.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:04.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:04.850 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:12:04.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:12:04.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:04.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:12:05.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:06.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:06.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:06.786 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:12:06.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:12:06.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:06.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:12:07.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:08.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:08.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:08.443 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:12:08.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:12:08.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:08.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:12:09.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:10.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:10.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:10.246 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:12:10.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:12:10.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:10.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:12:11.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:12.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:12.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:12.405 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:12:12.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:12:12.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:12.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:12:12.807 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/236392,v1:127.0.0.1:6803/236392] [v2:127.0.0.1:6804/236392,v1:127.0.0.1:6805/236392] exists,up 9676b58c-01f5-4bd2-afc0-67039d857939 2023-11-02T09:12:12.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:12:12.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:12:12.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:12:12.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:87: TEST_markdown_boot: run_osd td/osd-markdown 1 2023-11-02T09:12:12.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2023-11-02T09:12:12.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:12:12.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-11-02T09:12:12.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:12:12.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2023-11-02T09:12:12.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 ' 2023-11-02T09:12:12.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:12:12.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:12:12.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:12:12.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2023-11-02T09:12:12.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2023-11-02T09:12:12.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:12:12.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:12:12.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2023-11-02T09:12:12.815 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:12:12.815 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:12:12.817 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:12:12.817 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:12:12.817 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:12:12.818 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:12:12.818 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:12:12.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:12:12.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:12:12.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:12:12.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:12:12.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2023-11-02T09:12:12.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2023-11-02T09:12:12.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:12:12.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:12:12.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:12:12.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:12:12.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:12:12.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:12:12.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2023-11-02T09:12:12.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:12:12.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8d43b515-5ecd-4fc5-a6ad-bb370c6a90e0 2023-11-02T09:12:12.822 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 8d43b515-5ecd-4fc5-a6ad-bb370c6a90e0 2023-11-02T09:12:12.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8d43b515-5ecd-4fc5-a6ad-bb370c6a90e0' 2023-11-02T09:12:12.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:12:12.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDsZ0NlxeAmMRAA69NXwSPB7Tkq17znwFg7AQ== 2023-11-02T09:12:12.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDsZ0NlxeAmMRAA69NXwSPB7Tkq17znwFg7AQ=="}' 2023-11-02T09:12:12.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8d43b515-5ecd-4fc5-a6ad-bb370c6a90e0 -i td/osd-markdown/1/new.json 2023-11-02T09:12:13.326 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:12:13.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2023-11-02T09:12:13.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDsZ0NlxeAmMRAA69NXwSPB7Tkq17znwFg7AQ== --osd-uuid 8d43b515-5ecd-4fc5-a6ad-bb370c6a90e0 2023-11-02T09:12:13.363 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:13.359+0000 7f0f0e80e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:12:13.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:13.361+0000 7f0f0e80e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:12:13.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:13.362+0000 7f0f0e80e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:12:13.366 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:13.363+0000 7f0f0e80e500 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2023-11-02T09:12:13.366 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:13.363+0000 7f0f0e80e500 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2023-11-02T09:12:13.366 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:13.363+0000 7f0f0e80e500 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2023-11-02T09:12:13.416 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:13.414+0000 7f0f0e80e500 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2023-11-02T09:12:18.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2023-11-02T09:12:18.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:12:18.830 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2023-11-02T09:12:18.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-11-02T09:12:18.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:12:19.374 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T09:12:19.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-11-02T09:12:19.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:12:19.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:12:19.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:12:19.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:12:19.398 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:19.395+0000 7f99c430e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:12:19.400 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:19.397+0000 7f99c430e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:12:19.401 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:19.398+0000 7f99c430e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:12:19.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-11-02T09:12:19.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:12:19.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T09:12:19.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:12:19.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:12:19.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:19.745 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:12:19.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:12:19.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:19.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:12:19.926 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:19.923+0000 7f99c430e500 -1 Falling back to public interface 2023-11-02T09:12:20.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:21.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:21.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:21.161 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:12:21.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:12:21.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:21.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:12:21.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:22.190 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:22.187+0000 7f99c430e500 -1 osd.1 0 log_to_monitors true 2023-11-02T09:12:22.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:22.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:22.589 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:12:22.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:12:22.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:22.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:12:22.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:23.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:23.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:23.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:12:23.967 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:12:23.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:23.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:12:24.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:25.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:25.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:25.365 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:12:25.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:12:25.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:25.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:12:25.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:26.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:26.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:26.918 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:12:26.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:12:26.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:26.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:12:27.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:28.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:28.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:28.496 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:12:28.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:12:28.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:28.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:12:28.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:29.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:29.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:29.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:12:29.919 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:12:29.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:29.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:12:30.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:31.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:31.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:31.307 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:12:31.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:12:31.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:31.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:12:31.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:32.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:32.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:32.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:12:32.713 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:12:32.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:32.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:12:33.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:34.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:34.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:34.193 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:12:34.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:12:34.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:34.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:12:34.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:35.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:35.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:35.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:12:35.872 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:12:35.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:35.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:12:36.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:37.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:37.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:37.543 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:12:37.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:12:37.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:37.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:12:38.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:39.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:39.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:39.546 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:12:39.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:12:39.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:39.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:12:40.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:41.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:41.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:41.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:12:41.476 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:12:41.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:41.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:12:42.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:43.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:43.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:43.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:12:43.548 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:12:43.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:43.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:12:44.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:45.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:45.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:45.445 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:12:45.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:12:45.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:45.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:12:45.858 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/238105,v1:127.0.0.1:6811/238105] [v2:127.0.0.1:6812/238105,v1:127.0.0.1:6813/238105] exists,up 8d43b515-5ecd-4fc5-a6ad-bb370c6a90e0 2023-11-02T09:12:45.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:12:45.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:12:45.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:12:45.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:88: TEST_markdown_boot: run_osd td/osd-markdown 2 2023-11-02T09:12:45.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2023-11-02T09:12:45.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:12:45.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-11-02T09:12:45.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:12:45.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2023-11-02T09:12:45.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 ' 2023-11-02T09:12:45.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:12:45.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:12:45.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:12:45.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2023-11-02T09:12:45.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2023-11-02T09:12:45.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:12:45.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:12:45.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2023-11-02T09:12:45.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:12:45.869 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:12:45.869 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:12:45.869 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:12:45.869 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:12:45.870 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:12:45.870 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:12:45.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:12:45.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:12:45.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:12:45.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:12:45.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2023-11-02T09:12:45.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2023-11-02T09:12:45.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:12:45.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:12:45.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:12:45.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:12:45.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:12:45.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:12:45.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2023-11-02T09:12:45.874 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:12:45.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=569b6b97-d5b5-406c-8cc1-2656b2989285 2023-11-02T09:12:45.875 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 569b6b97-d5b5-406c-8cc1-2656b2989285 2023-11-02T09:12:45.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 569b6b97-d5b5-406c-8cc1-2656b2989285' 2023-11-02T09:12:45.875 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:12:45.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQANaENlCowdNBAA0evlAO88lFKYJle4kmOtwQ== 2023-11-02T09:12:45.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQANaENlCowdNBAA0evlAO88lFKYJle4kmOtwQ=="}' 2023-11-02T09:12:45.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 569b6b97-d5b5-406c-8cc1-2656b2989285 -i td/osd-markdown/2/new.json 2023-11-02T09:12:46.365 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:12:46.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2023-11-02T09:12:46.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQANaENlCowdNBAA0evlAO88lFKYJle4kmOtwQ== --osd-uuid 569b6b97-d5b5-406c-8cc1-2656b2989285 2023-11-02T09:12:46.402 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:46.399+0000 7f712ce0d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:12:46.404 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:46.401+0000 7f712ce0d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:12:46.405 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:46.402+0000 7f712ce0d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:12:46.405 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:46.402+0000 7f712ce0d500 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2023-11-02T09:12:46.405 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:46.402+0000 7f712ce0d500 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2023-11-02T09:12:46.406 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:46.402+0000 7f712ce0d500 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2023-11-02T09:12:46.473 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:46.470+0000 7f712ce0d500 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2023-11-02T09:12:52.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2023-11-02T09:12:52.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:12:52.039 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2023-11-02T09:12:52.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-11-02T09:12:52.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:12:52.551 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T09:12:52.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-11-02T09:12:52.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:12:52.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:12:52.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:12:52.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:12:52.574 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:52.571+0000 7fc5a2e1d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:12:52.576 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:52.573+0000 7fc5a2e1d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:12:52.577 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:52.574+0000 7fc5a2e1d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:12:52.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-11-02T09:12:52.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:12:52.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T09:12:52.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:12:52.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:12:52.960 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:12:52.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:52.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:12:52.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:52.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:12:53.104 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:53.101+0000 7fc5a2e1d500 -1 Falling back to public interface 2023-11-02T09:12:53.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:54.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:54.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:54.331 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:12:54.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:12:54.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:54.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:12:54.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:55.466 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:12:55.463+0000 7fc5a2e1d500 -1 osd.2 0 log_to_monitors true 2023-11-02T09:12:55.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:55.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:55.737 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:12:55.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:12:55.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:55.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:12:56.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:57.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:57.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:57.224 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:12:57.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:12:57.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:57.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:12:57.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:12:58.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:12:58.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:12:58.867 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:12:58.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:12:58.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:12:58.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:12:59.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:13:00.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:13:00.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:13:00.390 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:13:00.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:13:00.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:13:00.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:13:00.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:13:01.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:13:01.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:13:01.790 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:13:01.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:13:01.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:13:01.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:13:02.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:13:03.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:13:03.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:13:03.189 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:13:03.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:13:03.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:13:03.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:13:03.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:13:04.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:13:04.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:13:04.670 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:13:04.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:13:04.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:13:04.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:13:05.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:13:06.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:13:06.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:13:06.052 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:13:06.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:13:06.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:13:06.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:13:06.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:13:07.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:13:07.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:13:07.547 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:13:07.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:13:07.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:13:07.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:13:08.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:13:09.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:13:09.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:13:09.005 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:13:09.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:13:09.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:13:09.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:13:09.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:13:10.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:13:10.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:13:10.401 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:13:10.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:13:10.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:13:10.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:13:11.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:13:12.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:13:12.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:13:12.457 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:13:12.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:13:12.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:13:12.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:13:13.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:13:14.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:13:14.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:13:14.168 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:13:14.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:13:14.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:13:14.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:13:15.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:13:16.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:13:16.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:13:16.223 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:13:16.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:13:16.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:13:16.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:13:17.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:13:18.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:13:18.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:13:18.011 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:13:18.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:13:18.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:13:18.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:13:18.385 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/239818,v1:127.0.0.1:6819/239818] [v2:127.0.0.1:6820/239818,v1:127.0.0.1:6821/239818] exists,up 569b6b97-d5b5-406c-8cc1-2656b2989285 2023-11-02T09:13:18.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:13:18.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:13:18.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:13:18.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:90: TEST_markdown_boot: create_rbd_pool 2023-11-02T09:13:18.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-11-02T09:13:18.769 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' does not exist 2023-11-02T09:13:18.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-11-02T09:13:18.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-11-02T09:13:19.633 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' already exists 2023-11-02T09:13:19.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:13:20.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-11-02T09:13:21.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:93: TEST_markdown_boot: local count=3 2023-11-02T09:13:21.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:94: TEST_markdown_boot: local sleeptime=10 2023-11-02T09:13:21.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:95: TEST_markdown_boot: local period=120 2023-11-02T09:13:21.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:96: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2023-11-02T09:13:22.041 INFO:tasks.workunit.client.0.smithi116.stdout:{} 2023-11-02T09:13:22.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:97: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_period 120' 2023-11-02T09:13:22.153 INFO:tasks.workunit.client.0.smithi116.stdout:{} 2023-11-02T09:13:22.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:99: TEST_markdown_boot: markdown_N_impl 3 120 10 2023-11-02T09:13:22.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=3 2023-11-02T09:13:22.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=120 2023-11-02T09:13:22.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2023-11-02T09:13:22.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 3 2023-11-02T09:13:22.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2023-11-02T09:13:22.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2023-11-02T09:13:22.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2023-11-02T09:13:22.675 INFO:tasks.workunit.client.0.smithi116.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2023-11-02T09:13:22.675 INFO:tasks.workunit.client.0.smithi116.stdout:-1 0.29306 root default 2023-11-02T09:13:22.675 INFO:tasks.workunit.client.0.smithi116.stdout:-3 0.29306 host smithi116 2023-11-02T09:13:22.675 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:13:22.675 INFO:tasks.workunit.client.0.smithi116.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2023-11-02T09:13:22.676 INFO:tasks.workunit.client.0.smithi116.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2023-11-02T09:13:22.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2023-11-02T09:13:22.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2023-11-02T09:13:22.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2023-11-02T09:13:23.076 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:13:23.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2023-11-02T09:13:23.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2023-11-02T09:13:23.518 INFO:tasks.workunit.client.0.smithi116.stderr:marked down osd.0. 2023-11-02T09:13:23.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2023-11-02T09:13:33.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2023-11-02T09:13:33.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2023-11-02T09:13:33.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2023-11-02T09:13:34.047 INFO:tasks.workunit.client.0.smithi116.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2023-11-02T09:13:34.047 INFO:tasks.workunit.client.0.smithi116.stdout:-1 0.29306 root default 2023-11-02T09:13:34.048 INFO:tasks.workunit.client.0.smithi116.stdout:-3 0.29306 host smithi116 2023-11-02T09:13:34.048 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:13:34.048 INFO:tasks.workunit.client.0.smithi116.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2023-11-02T09:13:34.048 INFO:tasks.workunit.client.0.smithi116.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2023-11-02T09:13:34.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2023-11-02T09:13:34.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2023-11-02T09:13:34.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2023-11-02T09:13:34.429 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:13:34.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2023-11-02T09:13:34.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2023-11-02T09:13:34.816 INFO:tasks.workunit.client.0.smithi116.stderr:marked down osd.0. 2023-11-02T09:13:34.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2023-11-02T09:13:44.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2023-11-02T09:13:44.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2023-11-02T09:13:44.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2023-11-02T09:13:45.326 INFO:tasks.workunit.client.0.smithi116.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2023-11-02T09:13:45.326 INFO:tasks.workunit.client.0.smithi116.stdout:-1 0.29306 root default 2023-11-02T09:13:45.326 INFO:tasks.workunit.client.0.smithi116.stdout:-3 0.29306 host smithi116 2023-11-02T09:13:45.327 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:13:45.327 INFO:tasks.workunit.client.0.smithi116.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2023-11-02T09:13:45.327 INFO:tasks.workunit.client.0.smithi116.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2023-11-02T09:13:45.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2023-11-02T09:13:45.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2023-11-02T09:13:45.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2023-11-02T09:13:45.701 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:13:45.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2023-11-02T09:13:45.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2023-11-02T09:13:46.088 INFO:tasks.workunit.client.0.smithi116.stderr:marked down osd.0. 2023-11-02T09:13:46.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2023-11-02T09:13:56.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:101: TEST_markdown_boot: sleep 15 2023-11-02T09:14:11.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:102: TEST_markdown_boot: ceph tell osd.0 get_latest_osdmap 2023-11-02T09:14:11.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: ceph osd tree 2023-11-02T09:14:11.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep up 2023-11-02T09:14:11.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep osd.0 2023-11-02T09:14:11.630 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:14:11.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2023-11-02T09:14:11.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2023-11-02T09:14:11.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:14:11.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2023-11-02T09:14:11.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:14:11.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:14:11.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:14:11.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:14:11.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:14:11.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:14:11.738 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:14:11.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:14:11.739 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:14:11.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:14:11.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:14:11.741 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:14:11.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:14:11.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:14:11.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:14:11.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:14:11.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:14:11.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:14:11.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2023-11-02T09:14:11.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:14:11.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:14:11.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:14:11.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.235308 2023-11-02T09:14:11.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:14:11.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:14:11.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2023-11-02T09:14:11.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2023-11-02T09:14:11.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2023-11-02T09:14:11.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2023-11-02T09:14:11.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2023-11-02T09:14:11.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:14:11.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2023-11-02T09:14:11.761 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:14:11.761 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:14:11.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:14:11.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:14:11.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:14:11.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:14:11.764 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:14:11.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:14:11.764 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:14:11.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:14:11.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:14:11.765 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:14:11.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:14:11.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:14:11.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:14:11.766 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:14:11.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:14:11.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:14:11.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2023-11-02T09:14:11.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:14:11.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:14:11.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:14:11.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.235308 2023-11-02T09:14:11.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:14:11.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:14:11.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2023-11-02T09:14:11.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T09:14:11.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:14:11.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:14:11.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.235308 2023-11-02T09:14:11.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T09:14:11.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-11-02T09:14:11.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T09:14:11.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2023-11-02T09:14:11.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot_exceed_time td/osd-markdown 2023-11-02T09:14:11.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:107: TEST_markdown_boot_exceed_time: local dir=td/osd-markdown 2023-11-02T09:14:11.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:109: TEST_markdown_boot_exceed_time: run_mon td/osd-markdown a 2023-11-02T09:14:11.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2023-11-02T09:14:11.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T09:14:11.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T09:14:11.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T09:14:11.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2023-11-02T09:14:11.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2023-11-02T09:14:12.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T09:14:12.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:14:12.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:14:12.140 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:14:12.141 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:14:12.141 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:14:12.141 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:14:12.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T09:14:12.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T09:14:12.325 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T09:14:12.326 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:14:12.326 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:14:12.326 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T09:14:12.326 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T09:14:12.326 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:14:12.327 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:14:12.327 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:14:12.327 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:14:12.327 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:14:12.327 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:14:12.328 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.235308/ceph-mon.a.asok 2023-11-02T09:14:12.328 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:14:12.328 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.235308/ceph-mon.a.asok config get fsid 2023-11-02T09:14:12.405 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T09:14:12.406 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:14:12.406 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:14:12.406 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T09:14:12.406 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T09:14:12.407 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:14:12.407 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:14:12.407 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:14:12.407 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:14:12.407 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:14:12.408 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:14:12.408 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.235308/ceph-mon.a.asok 2023-11-02T09:14:12.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:14:12.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.235308/ceph-mon.a.asok config get mon_host 2023-11-02T09:14:12.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:110: TEST_markdown_boot_exceed_time: run_mgr td/osd-markdown x 2023-11-02T09:14:12.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2023-11-02T09:14:12.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T09:14:12.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T09:14:12.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T09:14:12.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2023-11-02T09:14:12.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T09:14:13.329 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T09:14:13.330 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:14:13.330 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:14:13.330 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:14:13.330 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:14:13.331 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:14:13.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:14:13.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:14:13.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:14:13.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:111: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 0 2023-11-02T09:14:13.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2023-11-02T09:14:13.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:14:13.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T09:14:13.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:14:13.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2023-11-02T09:14:13.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 ' 2023-11-02T09:14:13.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:14:13.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:14:13.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:14:13.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2023-11-02T09:14:13.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2023-11-02T09:14:13.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:14:13.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:14:13.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2023-11-02T09:14:13.361 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:14:13.361 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:14:13.361 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:14:13.362 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:14:13.362 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:14:13.362 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:14:13.362 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:14:13.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:14:13.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:14:13.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:14:13.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:14:13.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2023-11-02T09:14:13.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2023-11-02T09:14:13.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:14:13.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:14:13.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:14:13.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:14:13.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:14:13.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:14:13.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2023-11-02T09:14:13.368 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:14:13.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d86be4c9-cb3c-4ea5-8d00-744181a3e24d 2023-11-02T09:14:13.369 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 d86be4c9-cb3c-4ea5-8d00-744181a3e24d 2023-11-02T09:14:13.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d86be4c9-cb3c-4ea5-8d00-744181a3e24d' 2023-11-02T09:14:13.370 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:14:13.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBlaENlApocFhAAjtXby2O/xIqevWDI+ineBA== 2023-11-02T09:14:13.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBlaENlApocFhAAjtXby2O/xIqevWDI+ineBA=="}' 2023-11-02T09:14:13.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d86be4c9-cb3c-4ea5-8d00-744181a3e24d -i td/osd-markdown/0/new.json 2023-11-02T09:14:13.725 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:14:13.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2023-11-02T09:14:13.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBlaENlApocFhAAjtXby2O/xIqevWDI+ineBA== --osd-uuid d86be4c9-cb3c-4ea5-8d00-744181a3e24d 2023-11-02T09:14:13.761 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:13.758+0000 7f333faab500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:14:13.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:13.760+0000 7f333faab500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:14:13.763 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:13.761+0000 7f333faab500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:14:13.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:13.761+0000 7f333faab500 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2023-11-02T09:14:13.764 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:13.761+0000 7f333faab500 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2023-11-02T09:14:13.765 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:13.761+0000 7f333faab500 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2023-11-02T09:14:13.840 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:13.838+0000 7f333faab500 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2023-11-02T09:14:19.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2023-11-02T09:14:19.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:14:19.139 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T09:14:19.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T09:14:19.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:14:19.652 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T09:14:19.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T09:14:19.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:14:19.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:14:19.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:14:19.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:14:19.677 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:19.674+0000 7f42ab020500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:14:19.682 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:19.676+0000 7f42ab020500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:14:19.682 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:19.677+0000 7f42ab020500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:14:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T09:14:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:14:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T09:14:20.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:14:20.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:14:20.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:20.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:14:20.078 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:14:20.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:20.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:14:20.202 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:20.200+0000 7f42ab020500 -1 Falling back to public interface 2023-11-02T09:14:20.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:21.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:21.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:21.487 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:14:21.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:14:21.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:21.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:14:21.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:22.506 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:22.503+0000 7f42ab020500 -1 osd.0 0 log_to_monitors true 2023-11-02T09:14:22.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:22.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:22.904 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:14:22.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:14:22.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:22.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:14:23.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:24.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:24.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:24.411 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:14:24.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:14:24.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:24.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:14:24.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:25.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:25.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:25.795 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:14:25.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:14:25.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:25.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:14:26.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:27.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:27.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:27.202 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:14:27.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:14:27.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:27.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:14:27.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:28.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:28.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:28.582 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:14:28.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:14:28.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:28.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:14:28.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:29.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:29.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:29.964 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:14:29.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:14:29.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:29.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:14:30.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:31.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:31.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:31.344 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:14:31.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:14:31.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:31.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:14:31.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:32.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:32.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:32.744 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:14:32.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:14:32.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:32.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:14:33.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:34.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:34.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:34.167 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:14:34.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:14:34.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:34.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:14:34.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:35.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:35.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:35.767 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:14:35.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:14:35.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:35.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:14:36.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:37.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:37.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:37.253 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:14:37.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:14:37.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:37.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:14:38.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:39.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:39.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:39.233 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:14:39.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:14:39.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:39.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:14:39.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:40.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:40.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:40.850 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:14:40.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:14:40.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:40.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:14:41.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:42.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:42.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:42.519 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:14:42.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:14:42.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:42.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:14:43.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:44.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:44.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:44.582 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:14:44.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:14:44.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:44.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:14:45.083 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/242254,v1:127.0.0.1:6803/242254] [v2:127.0.0.1:6804/242254,v1:127.0.0.1:6805/242254] exists,up d86be4c9-cb3c-4ea5-8d00-744181a3e24d 2023-11-02T09:14:45.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:14:45.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:14:45.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:14:45.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:112: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 1 2023-11-02T09:14:45.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2023-11-02T09:14:45.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:14:45.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-11-02T09:14:45.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:14:45.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2023-11-02T09:14:45.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 ' 2023-11-02T09:14:45.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:14:45.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:14:45.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:14:45.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2023-11-02T09:14:45.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2023-11-02T09:14:45.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:14:45.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:14:45.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2023-11-02T09:14:45.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:14:45.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:14:45.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:14:45.091 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:14:45.091 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:14:45.091 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:14:45.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:14:45.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:14:45.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:14:45.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:14:45.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:14:45.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2023-11-02T09:14:45.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2023-11-02T09:14:45.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:14:45.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:14:45.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:14:45.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:14:45.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:14:45.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:14:45.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2023-11-02T09:14:45.094 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:14:45.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c0bcb103-435e-4f1e-b28d-90725c2c94d4 2023-11-02T09:14:45.095 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 c0bcb103-435e-4f1e-b28d-90725c2c94d4 2023-11-02T09:14:45.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c0bcb103-435e-4f1e-b28d-90725c2c94d4' 2023-11-02T09:14:45.095 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:14:45.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCFaENlgOHkBRAAmBjyqLZboaNZkRpkQDukeg== 2023-11-02T09:14:45.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCFaENlgOHkBRAAmBjyqLZboaNZkRpkQDukeg=="}' 2023-11-02T09:14:45.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c0bcb103-435e-4f1e-b28d-90725c2c94d4 -i td/osd-markdown/1/new.json 2023-11-02T09:14:45.585 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:14:45.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2023-11-02T09:14:45.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCFaENlgOHkBRAAmBjyqLZboaNZkRpkQDukeg== --osd-uuid c0bcb103-435e-4f1e-b28d-90725c2c94d4 2023-11-02T09:14:45.622 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:45.619+0000 7fd81aa15500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:14:45.623 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:45.620+0000 7fd81aa15500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:14:45.624 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:45.621+0000 7fd81aa15500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:14:45.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:45.622+0000 7fd81aa15500 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2023-11-02T09:14:45.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:45.622+0000 7fd81aa15500 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2023-11-02T09:14:45.625 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:45.622+0000 7fd81aa15500 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2023-11-02T09:14:45.693 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:45.689+0000 7fd81aa15500 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2023-11-02T09:14:51.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2023-11-02T09:14:51.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:14:51.069 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2023-11-02T09:14:51.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-11-02T09:14:51.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:14:51.583 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T09:14:51.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-11-02T09:14:51.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:14:51.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:14:51.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:14:51.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:14:51.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:51.604+0000 7f8678c13500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:14:51.609 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:51.606+0000 7f8678c13500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:14:51.610 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:51.607+0000 7f8678c13500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:14:52.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-11-02T09:14:52.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:14:52.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T09:14:52.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:14:52.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:14:52.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:52.003 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:14:52.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:14:52.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:52.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:14:52.149 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:52.147+0000 7f8678c13500 -1 Falling back to public interface 2023-11-02T09:14:52.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:53.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:53.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:53.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:14:53.422 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:14:53.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:53.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:14:53.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:54.441 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:14:54.438+0000 7f8678c13500 -1 osd.1 0 log_to_monitors true 2023-11-02T09:14:54.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:54.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:54.831 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:14:54.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:14:54.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:54.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:14:55.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:56.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:56.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:56.214 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:14:56.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:14:56.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:56.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:14:56.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:57.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:57.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:57.597 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:14:57.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:14:57.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:57.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:14:58.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:14:59.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:14:59.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:14:59.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:14:59.267 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:14:59.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:14:59.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:14:59.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:00.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:00.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:00.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:15:00.781 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:15:00.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:00.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:15:01.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:02.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:02.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:02.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:15:02.276 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:15:02.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:02.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:15:02.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:03.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:03.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:03.668 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:15:03.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:15:03.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:03.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:15:04.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:05.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:05.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:05.227 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:15:05.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:15:05.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:05.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:15:05.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:06.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:06.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:06.658 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:15:06.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:15:06.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:06.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:15:07.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:08.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:08.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:08.036 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:15:08.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:15:08.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:08.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:15:08.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:09.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:09.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:09.538 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:15:09.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:15:09.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:09.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:15:10.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:11.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:11.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:11.867 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:15:11.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:15:11.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:11.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:15:12.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:13.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:13.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:13.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:15:13.869 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:15:13.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:13.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:15:14.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:15.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:15.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:15.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:15:15.969 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:15:15.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:15.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:15:16.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:17.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:17.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:17.345 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:15:17.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:15:17.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:17.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:15:17.754 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/243967,v1:127.0.0.1:6811/243967] [v2:127.0.0.1:6812/243967,v1:127.0.0.1:6813/243967] exists,up c0bcb103-435e-4f1e-b28d-90725c2c94d4 2023-11-02T09:15:17.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:15:17.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:15:17.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:15:17.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:113: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 2 2023-11-02T09:15:17.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2023-11-02T09:15:17.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:15:17.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-11-02T09:15:17.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:15:17.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2023-11-02T09:15:17.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 ' 2023-11-02T09:15:17.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:15:17.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:15:17.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:15:17.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2023-11-02T09:15:17.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2023-11-02T09:15:17.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:15:17.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:15:17.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2023-11-02T09:15:17.763 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:15:17.763 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:15:17.763 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:15:17.763 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:15:17.764 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:15:17.764 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:15:17.764 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:15:17.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:15:17.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:15:17.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:15:17.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:15:17.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2023-11-02T09:15:17.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2023-11-02T09:15:17.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:15:17.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:15:17.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:15:17.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:15:17.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:15:17.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:15:17.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2023-11-02T09:15:17.767 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:15:17.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2ac637c2-0234-432d-8608-1d40fa816657 2023-11-02T09:15:17.769 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 2ac637c2-0234-432d-8608-1d40fa816657 2023-11-02T09:15:17.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2ac637c2-0234-432d-8608-1d40fa816657' 2023-11-02T09:15:17.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:15:17.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQClaENlHtPzLRAAO68j74OKqbESbgt6bqPq2w== 2023-11-02T09:15:17.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQClaENlHtPzLRAAO68j74OKqbESbgt6bqPq2w=="}' 2023-11-02T09:15:17.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2ac637c2-0234-432d-8608-1d40fa816657 -i td/osd-markdown/2/new.json 2023-11-02T09:15:18.262 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:15:18.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2023-11-02T09:15:18.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQClaENlHtPzLRAAO68j74OKqbESbgt6bqPq2w== --osd-uuid 2ac637c2-0234-432d-8608-1d40fa816657 2023-11-02T09:15:18.299 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:15:18.296+0000 7fe901616500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:15:18.301 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:15:18.298+0000 7fe901616500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:15:18.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:15:18.299+0000 7fe901616500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:15:18.302 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:15:18.299+0000 7fe901616500 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2023-11-02T09:15:18.303 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:15:18.299+0000 7fe901616500 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2023-11-02T09:15:18.303 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:15:18.299+0000 7fe901616500 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2023-11-02T09:15:18.336 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:15:18.333+0000 7fe901616500 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2023-11-02T09:15:24.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2023-11-02T09:15:24.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:15:24.217 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2023-11-02T09:15:24.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-11-02T09:15:24.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:15:24.727 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T09:15:24.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-11-02T09:15:24.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:15:24.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:15:24.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:15:24.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:15:24.752 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:15:24.749+0000 7f3305611500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:15:24.776 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:15:24.773+0000 7f3305611500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:15:24.777 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:15:24.775+0000 7f3305611500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:15:25.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-11-02T09:15:25.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:15:25.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T09:15:25.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:15:25.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:15:25.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:25.137 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:15:25.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:15:25.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:25.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:15:25.313 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:15:25.311+0000 7f3305611500 -1 Falling back to public interface 2023-11-02T09:15:25.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:26.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:26.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:26.509 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:15:26.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:15:26.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:26.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:15:26.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:27.694 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:15:27.692+0000 7f3305611500 -1 osd.2 0 log_to_monitors true 2023-11-02T09:15:27.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:27.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:27.928 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:15:27.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:15:27.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:27.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:15:28.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:29.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:29.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:29.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:15:29.390 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:15:29.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:29.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:15:29.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:30.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:30.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:30.887 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:15:30.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:15:30.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:30.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:15:31.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:32.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:32.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:32.284 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:15:32.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:15:32.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:32.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:15:32.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:33.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:33.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:33.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:15:33.798 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:15:33.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:33.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:15:34.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:35.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:35.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:35.288 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:15:35.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:15:35.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:35.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:15:35.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:36.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:36.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:36.710 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:15:36.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:15:36.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:36.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:15:37.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:38.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:38.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:38.096 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:15:38.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:15:38.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:38.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:15:38.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:39.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:39.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:39.477 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:15:39.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:15:39.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:39.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:15:40.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:41.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:41.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:41.288 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:15:41.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:15:41.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:41.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:15:42.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:43.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:43.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:43.253 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:15:43.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:15:43.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:43.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:15:43.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:44.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:44.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:44.987 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:15:44.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:15:44.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:44.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:15:45.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:46.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:46.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:46.757 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:15:46.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:15:46.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:46.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:15:47.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:48.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:48.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:48.832 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:15:48.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:15:48.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:48.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:15:49.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:15:50.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:15:50.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:15:50.838 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:15:50.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:15:50.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:15:50.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:15:51.256 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/245680,v1:127.0.0.1:6819/245680] [v2:127.0.0.1:6820/245680,v1:127.0.0.1:6821/245680] exists,up 2ac637c2-0234-432d-8608-1d40fa816657 2023-11-02T09:15:51.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:15:51.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:15:51.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:15:51.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:115: TEST_markdown_boot_exceed_time: create_rbd_pool 2023-11-02T09:15:51.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-11-02T09:15:51.632 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' does not exist 2023-11-02T09:15:51.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-11-02T09:15:51.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-11-02T09:15:52.168 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' already exists 2023-11-02T09:15:52.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:15:53.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-11-02T09:15:54.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:118: TEST_markdown_boot_exceed_time: local count=3 2023-11-02T09:15:54.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:119: TEST_markdown_boot_exceed_time: local period=20 2023-11-02T09:15:54.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:120: TEST_markdown_boot_exceed_time: local sleeptime=10 2023-11-02T09:15:54.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:121: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2023-11-02T09:15:54.486 INFO:tasks.workunit.client.0.smithi116.stdout:{} 2023-11-02T09:15:54.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:122: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_period 20' 2023-11-02T09:15:54.599 INFO:tasks.workunit.client.0.smithi116.stdout:{} 2023-11-02T09:15:54.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:124: TEST_markdown_boot_exceed_time: markdown_N_impl 4 20 10 2023-11-02T09:15:54.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2023-11-02T09:15:54.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=20 2023-11-02T09:15:54.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2023-11-02T09:15:54.610 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2023-11-02T09:15:54.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2023-11-02T09:15:54.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2023-11-02T09:15:54.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2023-11-02T09:15:55.114 INFO:tasks.workunit.client.0.smithi116.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2023-11-02T09:15:55.114 INFO:tasks.workunit.client.0.smithi116.stdout:-1 0.29306 root default 2023-11-02T09:15:55.114 INFO:tasks.workunit.client.0.smithi116.stdout:-3 0.29306 host smithi116 2023-11-02T09:15:55.115 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:15:55.115 INFO:tasks.workunit.client.0.smithi116.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2023-11-02T09:15:55.115 INFO:tasks.workunit.client.0.smithi116.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2023-11-02T09:15:55.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2023-11-02T09:15:55.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2023-11-02T09:15:55.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2023-11-02T09:15:55.528 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:15:55.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2023-11-02T09:15:55.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2023-11-02T09:15:55.918 INFO:tasks.workunit.client.0.smithi116.stderr:marked down osd.0. 2023-11-02T09:15:55.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2023-11-02T09:16:05.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2023-11-02T09:16:05.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2023-11-02T09:16:06.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2023-11-02T09:16:06.436 INFO:tasks.workunit.client.0.smithi116.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2023-11-02T09:16:06.436 INFO:tasks.workunit.client.0.smithi116.stdout:-1 0.29306 root default 2023-11-02T09:16:06.437 INFO:tasks.workunit.client.0.smithi116.stdout:-3 0.29306 host smithi116 2023-11-02T09:16:06.437 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:16:06.437 INFO:tasks.workunit.client.0.smithi116.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2023-11-02T09:16:06.437 INFO:tasks.workunit.client.0.smithi116.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2023-11-02T09:16:06.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2023-11-02T09:16:06.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2023-11-02T09:16:06.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2023-11-02T09:16:06.848 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:16:06.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2023-11-02T09:16:06.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2023-11-02T09:16:07.246 INFO:tasks.workunit.client.0.smithi116.stderr:marked down osd.0. 2023-11-02T09:16:07.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2023-11-02T09:16:17.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2023-11-02T09:16:17.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2023-11-02T09:16:17.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2023-11-02T09:16:17.779 INFO:tasks.workunit.client.0.smithi116.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2023-11-02T09:16:17.780 INFO:tasks.workunit.client.0.smithi116.stdout:-1 0.29306 root default 2023-11-02T09:16:17.780 INFO:tasks.workunit.client.0.smithi116.stdout:-3 0.29306 host smithi116 2023-11-02T09:16:17.780 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:16:17.780 INFO:tasks.workunit.client.0.smithi116.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2023-11-02T09:16:17.781 INFO:tasks.workunit.client.0.smithi116.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2023-11-02T09:16:17.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2023-11-02T09:16:17.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2023-11-02T09:16:17.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2023-11-02T09:16:18.201 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:16:18.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2023-11-02T09:16:18.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2023-11-02T09:16:18.592 INFO:tasks.workunit.client.0.smithi116.stderr:marked down osd.0. 2023-11-02T09:16:18.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2023-11-02T09:16:28.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2023-11-02T09:16:28.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2023-11-02T09:16:28.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2023-11-02T09:16:29.116 INFO:tasks.workunit.client.0.smithi116.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2023-11-02T09:16:29.117 INFO:tasks.workunit.client.0.smithi116.stdout:-1 0.29306 root default 2023-11-02T09:16:29.117 INFO:tasks.workunit.client.0.smithi116.stdout:-3 0.29306 host smithi116 2023-11-02T09:16:29.117 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:16:29.117 INFO:tasks.workunit.client.0.smithi116.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2023-11-02T09:16:29.118 INFO:tasks.workunit.client.0.smithi116.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2023-11-02T09:16:29.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2023-11-02T09:16:29.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2023-11-02T09:16:29.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2023-11-02T09:16:29.505 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:16:29.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2023-11-02T09:16:29.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2023-11-02T09:16:29.945 INFO:tasks.workunit.client.0.smithi116.stderr:marked down osd.0. 2023-11-02T09:16:29.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2023-11-02T09:16:39.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:125: TEST_markdown_boot_exceed_time: sleep 15 2023-11-02T09:16:54.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:126: TEST_markdown_boot_exceed_time: ceph tell osd.0 get_latest_osdmap 2023-11-02T09:16:55.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: ceph osd tree 2023-11-02T09:16:55.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep up 2023-11-02T09:16:55.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep osd.0 2023-11-02T09:16:55.454 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:16:55.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2023-11-02T09:16:55.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2023-11-02T09:16:55.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:16:55.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2023-11-02T09:16:55.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:16:55.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:16:55.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:16:55.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:16:55.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:16:55.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:16:55.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:16:55.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:16:55.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:16:55.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:16:55.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:16:55.565 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:16:55.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:16:55.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:16:55.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:16:55.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:16:55.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:16:55.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:16:55.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2023-11-02T09:16:55.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:16:55.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:16:55.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:16:55.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.235308 2023-11-02T09:16:55.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:16:55.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:16:55.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2023-11-02T09:16:55.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2023-11-02T09:16:55.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2023-11-02T09:16:55.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2023-11-02T09:16:55.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2023-11-02T09:16:55.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:16:55.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2023-11-02T09:16:55.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:16:55.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:16:55.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:16:55.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:16:55.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:16:55.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:16:55.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:16:55.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:16:55.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:16:55.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:16:55.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:16:55.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:16:55.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:16:55.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:16:55.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:16:55.592 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:16:55.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:16:55.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:16:55.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2023-11-02T09:16:55.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:16:55.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:16:55.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:16:55.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.235308 2023-11-02T09:16:55.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:16:55.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:16:55.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2023-11-02T09:16:55.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T09:16:55.597 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:16:55.597 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:16:55.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.235308 2023-11-02T09:16:55.597 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T09:16:55.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-11-02T09:16:55.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T09:16:55.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2023-11-02T09:16:55.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_exceed_maxdown_count td/osd-markdown 2023-11-02T09:16:55.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:59: TEST_markdown_exceed_maxdown_count: local dir=td/osd-markdown 2023-11-02T09:16:55.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:61: TEST_markdown_exceed_maxdown_count: run_mon td/osd-markdown a 2023-11-02T09:16:55.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2023-11-02T09:16:55.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T09:16:55.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T09:16:55.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T09:16:55.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2023-11-02T09:16:55.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2023-11-02T09:16:55.879 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T09:16:55.879 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:16:55.880 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:16:55.880 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:16:55.880 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:16:55.880 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:16:55.880 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:16:55.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T09:16:56.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T09:16:56.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T09:16:56.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:16:56.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:16:56.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T09:16:56.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T09:16:56.057 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:16:56.057 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:16:56.057 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:16:56.057 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:16:56.058 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:16:56.058 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:16:56.058 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.235308/ceph-mon.a.asok 2023-11-02T09:16:56.059 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:16:56.059 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.235308/ceph-mon.a.asok config get fsid 2023-11-02T09:16:56.134 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T09:16:56.135 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:16:56.135 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:16:56.135 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T09:16:56.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T09:16:56.136 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:16:56.136 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:16:56.137 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:16:56.137 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:16:56.137 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:16:56.139 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:16:56.140 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.235308/ceph-mon.a.asok 2023-11-02T09:16:56.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:16:56.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.235308/ceph-mon.a.asok config get mon_host 2023-11-02T09:16:56.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:62: TEST_markdown_exceed_maxdown_count: run_mgr td/osd-markdown x 2023-11-02T09:16:56.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2023-11-02T09:16:56.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T09:16:56.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T09:16:56.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T09:16:56.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2023-11-02T09:16:56.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T09:16:57.058 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T09:16:57.058 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:16:57.059 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:16:57.059 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:16:57.059 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:16:57.060 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:16:57.060 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:16:57.060 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:16:57.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:16:57.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:63: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 0 2023-11-02T09:16:57.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2023-11-02T09:16:57.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:16:57.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T09:16:57.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:16:57.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2023-11-02T09:16:57.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 ' 2023-11-02T09:16:57.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:16:57.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:16:57.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:16:57.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2023-11-02T09:16:57.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2023-11-02T09:16:57.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:16:57.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:16:57.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2023-11-02T09:16:57.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:16:57.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:16:57.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:16:57.096 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:16:57.097 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:16:57.097 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:16:57.097 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:16:57.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:16:57.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:16:57.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:16:57.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:16:57.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2023-11-02T09:16:57.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2023-11-02T09:16:57.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:16:57.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:16:57.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:16:57.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:16:57.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:16:57.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:16:57.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2023-11-02T09:16:57.100 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:16:57.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d805f578-d4f2-439b-ad2e-f76bbbb3db65 2023-11-02T09:16:57.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d805f578-d4f2-439b-ad2e-f76bbbb3db65' 2023-11-02T09:16:57.101 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 d805f578-d4f2-439b-ad2e-f76bbbb3db65 2023-11-02T09:16:57.101 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:16:57.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAJaUNlum7sBRAAf2BKF9nfgAIoPZofzqnRcA== 2023-11-02T09:16:57.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAJaUNlum7sBRAAf2BKF9nfgAIoPZofzqnRcA=="}' 2023-11-02T09:16:57.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d805f578-d4f2-439b-ad2e-f76bbbb3db65 -i td/osd-markdown/0/new.json 2023-11-02T09:16:57.444 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:16:57.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2023-11-02T09:16:57.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAJaUNlum7sBRAAf2BKF9nfgAIoPZofzqnRcA== --osd-uuid d805f578-d4f2-439b-ad2e-f76bbbb3db65 2023-11-02T09:16:57.481 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:16:57.478+0000 7f72f9cb2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:16:57.483 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:16:57.480+0000 7f72f9cb2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:16:57.484 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:16:57.481+0000 7f72f9cb2500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:16:57.484 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:16:57.481+0000 7f72f9cb2500 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2023-11-02T09:16:57.484 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:16:57.481+0000 7f72f9cb2500 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2023-11-02T09:16:57.485 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:16:57.481+0000 7f72f9cb2500 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2023-11-02T09:16:57.527 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:16:57.524+0000 7f72f9cb2500 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2023-11-02T09:17:02.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2023-11-02T09:17:02.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:17:02.912 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T09:17:02.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T09:17:02.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:17:03.426 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T09:17:03.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T09:17:03.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:17:03.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:17:03.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:17:03.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:17:03.450 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:17:03.447+0000 7f4691f17500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:17:03.476 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:17:03.473+0000 7f4691f17500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:17:03.476 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:17:03.474+0000 7f4691f17500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:17:03.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T09:17:03.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:17:03.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T09:17:03.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:17:03.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:17:03.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:03.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:17:03.874 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:17:03.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:03.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:17:04.009 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:17:04.007+0000 7f4691f17500 -1 Falling back to public interface 2023-11-02T09:17:04.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:05.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:05.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:05.293 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:17:05.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:17:05.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:05.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:17:05.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:06.294 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:17:06.291+0000 7f4691f17500 -1 osd.0 0 log_to_monitors true 2023-11-02T09:17:06.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:06.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:06.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:17:06.673 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:17:06.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:06.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:17:07.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:08.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:08.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:08.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:17:08.178 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:17:08.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:08.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:17:08.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:09.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:09.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:09.565 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:17:09.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:17:09.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:09.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:17:09.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:10.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:10.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:10.975 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:17:10.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:17:10.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:10.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:17:11.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:12.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:12.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:12.379 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:17:12.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:17:12.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:12.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:17:12.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:13.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:13.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:13.754 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:17:13.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:17:13.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:13.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:17:14.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:15.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:15.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:15.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:17:15.253 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:17:15.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:15.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:17:15.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:16.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:16.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:16.658 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:17:16.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:17:16.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:16.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:17:17.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:18.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:18.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:18.123 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:17:18.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:17:18.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:18.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:17:18.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:19.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:19.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:19.901 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:17:19.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:17:19.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:19.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:17:20.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:21.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:21.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:21.569 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:17:21.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:17:21.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:21.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:17:22.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:23.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:23.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:23.352 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:17:23.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:17:23.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:23.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:17:23.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:24.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:24.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:24.961 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:17:24.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:17:24.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:24.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:17:25.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:26.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:26.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:26.423 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:17:26.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:17:26.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:26.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:17:27.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:28.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:28.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:28.397 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:17:28.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:17:28.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:28.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:17:28.910 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/248211,v1:127.0.0.1:6803/248211] [v2:127.0.0.1:6804/248211,v1:127.0.0.1:6805/248211] exists,up d805f578-d4f2-439b-ad2e-f76bbbb3db65 2023-11-02T09:17:28.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:17:28.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:17:28.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:17:28.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:64: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 1 2023-11-02T09:17:28.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2023-11-02T09:17:28.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:17:28.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-11-02T09:17:28.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:17:28.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2023-11-02T09:17:28.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 ' 2023-11-02T09:17:28.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:17:28.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:17:28.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:17:28.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2023-11-02T09:17:28.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2023-11-02T09:17:28.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:17:28.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:17:28.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2023-11-02T09:17:28.918 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:17:28.918 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:17:28.918 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:17:28.918 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:17:28.918 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:17:28.919 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:17:28.919 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:17:28.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:17:28.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:17:28.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:17:28.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:17:28.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2023-11-02T09:17:28.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2023-11-02T09:17:28.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:17:28.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:17:28.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:17:28.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:17:28.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:17:28.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:17:28.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2023-11-02T09:17:28.922 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:17:28.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5119216c-7c7c-4b0a-9470-494fce30c5ff 2023-11-02T09:17:28.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5119216c-7c7c-4b0a-9470-494fce30c5ff' 2023-11-02T09:17:28.923 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 5119216c-7c7c-4b0a-9470-494fce30c5ff 2023-11-02T09:17:28.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:17:28.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAoaUNlklM6NxAANOcFn7kR/mtbtTuLaJLrRQ== 2023-11-02T09:17:28.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAoaUNlklM6NxAANOcFn7kR/mtbtTuLaJLrRQ=="}' 2023-11-02T09:17:28.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5119216c-7c7c-4b0a-9470-494fce30c5ff -i td/osd-markdown/1/new.json 2023-11-02T09:17:29.448 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:17:29.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2023-11-02T09:17:29.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAoaUNlklM6NxAANOcFn7kR/mtbtTuLaJLrRQ== --osd-uuid 5119216c-7c7c-4b0a-9470-494fce30c5ff 2023-11-02T09:17:29.485 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:17:29.482+0000 7fbba340c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:17:29.487 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:17:29.484+0000 7fbba340c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:17:29.488 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:17:29.485+0000 7fbba340c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:17:29.488 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:17:29.485+0000 7fbba340c500 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2023-11-02T09:17:29.488 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:17:29.485+0000 7fbba340c500 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2023-11-02T09:17:29.488 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:17:29.485+0000 7fbba340c500 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2023-11-02T09:17:29.558 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:17:29.555+0000 7fbba340c500 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2023-11-02T09:17:35.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2023-11-02T09:17:35.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:17:35.224 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2023-11-02T09:17:35.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-11-02T09:17:35.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:17:35.738 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T09:17:35.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-11-02T09:17:35.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:17:35.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:17:35.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:17:35.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:17:35.762 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:17:35.759+0000 7f66201e0500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:17:35.779 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:17:35.776+0000 7f66201e0500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:17:35.780 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:17:35.778+0000 7f66201e0500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:17:36.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-11-02T09:17:36.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:17:36.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T09:17:36.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:17:36.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:17:36.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:36.194 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:17:36.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:17:36.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:36.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:17:36.315 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:17:36.313+0000 7f66201e0500 -1 Falling back to public interface 2023-11-02T09:17:36.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:37.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:37.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:37.561 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:17:37.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:17:37.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:37.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:17:37.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:38.566 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:17:38.563+0000 7f66201e0500 -1 osd.1 0 log_to_monitors true 2023-11-02T09:17:38.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:38.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:38.937 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:17:38.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:17:38.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:38.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:17:39.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:40.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:40.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:40.411 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:17:40.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:17:40.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:40.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:17:40.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:41.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:41.801 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:17:41.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:41.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:17:41.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:41.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:17:42.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:43.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:43.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:43.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:17:43.273 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:17:43.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:43.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:17:43.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:44.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:44.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:44.697 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:17:44.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:17:44.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:44.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:17:45.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:46.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:46.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:46.070 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:17:46.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:17:46.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:46.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:17:46.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:47.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:47.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:47.451 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:17:47.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:17:47.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:47.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:17:47.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:48.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:48.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:48.827 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:17:48.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:17:48.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:48.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:17:49.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:50.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:50.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:50.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:17:50.207 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:17:50.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:50.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:17:50.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:51.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:51.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:51.586 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:17:51.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:17:51.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:51.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:17:52.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:53.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:53.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:53.372 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:17:53.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:17:53.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:53.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:17:53.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:54.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:54.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:54.913 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:17:54.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:17:54.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:54.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:17:55.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:56.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:56.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:56.607 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:17:56.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:17:56.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:56.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:17:57.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:17:58.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:17:58.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:17:58.534 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:17:58.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:17:58.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:17:58.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:17:59.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:18:00.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:18:00.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:18:00.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:18:00.248 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:18:00.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:18:00.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:18:00.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:18:01.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:18:01.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:18:01.745 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T09:18:01.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T09:18:01.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:18:01.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:18:02.321 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/249925,v1:127.0.0.1:6811/249925] [v2:127.0.0.1:6812/249925,v1:127.0.0.1:6813/249925] exists,up 5119216c-7c7c-4b0a-9470-494fce30c5ff 2023-11-02T09:18:02.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:18:02.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:18:02.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:18:02.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:65: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 2 2023-11-02T09:18:02.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2023-11-02T09:18:02.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:18:02.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-11-02T09:18:02.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:18:02.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2023-11-02T09:18:02.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 ' 2023-11-02T09:18:02.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:18:02.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:18:02.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:18:02.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2023-11-02T09:18:02.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2023-11-02T09:18:02.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:18:02.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:18:02.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2023-11-02T09:18:02.330 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:18:02.330 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:18:02.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:18:02.331 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:18:02.331 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:18:02.331 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:18:02.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:18:02.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:18:02.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:18:02.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:18:02.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:18:02.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2023-11-02T09:18:02.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2023-11-02T09:18:02.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:18:02.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:18:02.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:18:02.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:18:02.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:18:02.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:18:02.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2023-11-02T09:18:02.334 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:18:02.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fb7cbf84-dae8-4ef4-99ff-d1ad24a8421b 2023-11-02T09:18:02.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 fb7cbf84-dae8-4ef4-99ff-d1ad24a8421b' 2023-11-02T09:18:02.335 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 fb7cbf84-dae8-4ef4-99ff-d1ad24a8421b 2023-11-02T09:18:02.336 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:18:02.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBKaUNlEJAyFBAAC6k20MTTzPoxkHeAhwaBZw== 2023-11-02T09:18:02.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBKaUNlEJAyFBAAC6k20MTTzPoxkHeAhwaBZw=="}' 2023-11-02T09:18:02.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fb7cbf84-dae8-4ef4-99ff-d1ad24a8421b -i td/osd-markdown/2/new.json 2023-11-02T09:18:02.839 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:18:02.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2023-11-02T09:18:02.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBKaUNlEJAyFBAAC6k20MTTzPoxkHeAhwaBZw== --osd-uuid fb7cbf84-dae8-4ef4-99ff-d1ad24a8421b 2023-11-02T09:18:02.876 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:18:02.873+0000 7f6be7715500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:18:02.877 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:18:02.875+0000 7f6be7715500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:18:02.878 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:18:02.876+0000 7f6be7715500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:18:02.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:18:02.876+0000 7f6be7715500 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2023-11-02T09:18:02.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:18:02.876+0000 7f6be7715500 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2023-11-02T09:18:02.879 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:18:02.876+0000 7f6be7715500 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2023-11-02T09:18:02.921 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:18:02.918+0000 7f6be7715500 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2023-11-02T09:18:08.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2023-11-02T09:18:08.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:18:08.471 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2023-11-02T09:18:08.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-11-02T09:18:08.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:18:08.984 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T09:18:08.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-11-02T09:18:08.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:18:08.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:18:08.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:18:08.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:18:09.008 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:18:09.005+0000 7f674c8cf500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:18:09.026 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:18:09.023+0000 7f674c8cf500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:18:09.027 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:18:09.024+0000 7f674c8cf500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:18:09.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-11-02T09:18:09.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:18:09.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T09:18:09.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:18:09.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:18:09.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:18:09.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:18:09.413 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:18:09.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:18:09.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:18:09.562 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:18:09.560+0000 7f674c8cf500 -1 Falling back to public interface 2023-11-02T09:18:09.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:18:10.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:18:10.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:18:10.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:18:10.779 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:18:10.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:18:10.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:18:11.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:18:11.854 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:18:11.851+0000 7f674c8cf500 -1 osd.2 0 log_to_monitors true 2023-11-02T09:18:12.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:18:12.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:18:12.185 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:18:12.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:18:12.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:18:12.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:18:12.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:18:13.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:18:13.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:18:13.639 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:18:13.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:18:13.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:18:13.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:18:14.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:18:15.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:18:15.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:18:15.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:18:15.024 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:18:15.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:18:15.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:18:15.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:18:16.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:18:16.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:18:16.470 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:18:16.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:18:16.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:18:16.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:18:16.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:18:17.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:18:17.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:18:17.992 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:18:17.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:18:17.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:18:17.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:18:18.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:18:19.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:18:19.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:18:19.389 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:18:19.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:18:19.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:18:19.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:18:19.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:18:20.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:18:20.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:18:20.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:18:20.887 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:18:20.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:18:20.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:18:21.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:18:22.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:18:22.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:18:22.382 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:18:22.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:18:22.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:18:22.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:18:22.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:18:23.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:18:23.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:18:23.781 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:18:23.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:18:23.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:18:23.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:18:24.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:18:25.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:18:25.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:18:25.450 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:18:25.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:18:25.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:18:25.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:18:26.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:18:27.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:18:27.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:18:27.347 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:18:27.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:18:27.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:18:27.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:18:28.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:18:29.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:18:29.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:18:29.205 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:18:29.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:18:29.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:18:29.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:18:29.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:18:30.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:18:30.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:18:30.950 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:18:30.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:18:30.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:18:30.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:18:31.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:18:32.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:18:32.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:18:32.739 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:18:32.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:18:32.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:18:32.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:18:33.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:18:34.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:18:34.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:18:34.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:18:34.455 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:18:34.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:18:34.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:18:34.836 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/251663,v1:127.0.0.1:6819/251663] [v2:127.0.0.1:6820/251663,v1:127.0.0.1:6821/251663] exists,up fb7cbf84-dae8-4ef4-99ff-d1ad24a8421b 2023-11-02T09:18:34.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:18:34.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:18:34.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:18:34.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:67: TEST_markdown_exceed_maxdown_count: create_rbd_pool 2023-11-02T09:18:34.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-11-02T09:18:35.304 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' does not exist 2023-11-02T09:18:35.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-11-02T09:18:35.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-11-02T09:18:35.998 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' already exists 2023-11-02T09:18:36.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:18:37.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-11-02T09:18:38.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:70: TEST_markdown_exceed_maxdown_count: local count=3 2023-11-02T09:18:38.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:71: TEST_markdown_exceed_maxdown_count: local sleeptime=10 2023-11-02T09:18:38.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:72: TEST_markdown_exceed_maxdown_count: local period=300 2023-11-02T09:18:38.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:73: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2023-11-02T09:18:39.107 INFO:tasks.workunit.client.0.smithi116.stdout:{} 2023-11-02T09:18:39.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:74: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_period 300' 2023-11-02T09:18:39.219 INFO:tasks.workunit.client.0.smithi116.stdout:{} 2023-11-02T09:18:39.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:76: TEST_markdown_exceed_maxdown_count: markdown_N_impl 4 300 10 2023-11-02T09:18:39.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2023-11-02T09:18:39.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=300 2023-11-02T09:18:39.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2023-11-02T09:18:39.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2023-11-02T09:18:39.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2023-11-02T09:18:39.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2023-11-02T09:18:39.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2023-11-02T09:18:39.692 INFO:tasks.workunit.client.0.smithi116.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2023-11-02T09:18:39.692 INFO:tasks.workunit.client.0.smithi116.stdout:-1 0.29306 root default 2023-11-02T09:18:39.692 INFO:tasks.workunit.client.0.smithi116.stdout:-3 0.29306 host smithi116 2023-11-02T09:18:39.693 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:18:39.693 INFO:tasks.workunit.client.0.smithi116.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2023-11-02T09:18:39.693 INFO:tasks.workunit.client.0.smithi116.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2023-11-02T09:18:39.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2023-11-02T09:18:39.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2023-11-02T09:18:39.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2023-11-02T09:18:40.107 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:18:40.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2023-11-02T09:18:40.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2023-11-02T09:18:40.472 INFO:tasks.workunit.client.0.smithi116.stderr:marked down osd.0. 2023-11-02T09:18:40.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2023-11-02T09:18:50.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2023-11-02T09:18:50.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2023-11-02T09:18:50.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2023-11-02T09:18:50.955 INFO:tasks.workunit.client.0.smithi116.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2023-11-02T09:18:50.955 INFO:tasks.workunit.client.0.smithi116.stdout:-1 0.29306 root default 2023-11-02T09:18:50.956 INFO:tasks.workunit.client.0.smithi116.stdout:-3 0.29306 host smithi116 2023-11-02T09:18:50.956 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:18:50.956 INFO:tasks.workunit.client.0.smithi116.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2023-11-02T09:18:50.956 INFO:tasks.workunit.client.0.smithi116.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2023-11-02T09:18:50.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2023-11-02T09:18:50.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2023-11-02T09:18:50.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2023-11-02T09:18:51.347 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:18:51.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2023-11-02T09:18:51.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2023-11-02T09:18:51.776 INFO:tasks.workunit.client.0.smithi116.stderr:marked down osd.0. 2023-11-02T09:18:51.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2023-11-02T09:19:01.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2023-11-02T09:19:01.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2023-11-02T09:19:01.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2023-11-02T09:19:02.307 INFO:tasks.workunit.client.0.smithi116.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2023-11-02T09:19:02.308 INFO:tasks.workunit.client.0.smithi116.stdout:-1 0.29306 root default 2023-11-02T09:19:02.308 INFO:tasks.workunit.client.0.smithi116.stdout:-3 0.29306 host smithi116 2023-11-02T09:19:02.308 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:19:02.308 INFO:tasks.workunit.client.0.smithi116.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2023-11-02T09:19:02.309 INFO:tasks.workunit.client.0.smithi116.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2023-11-02T09:19:02.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2023-11-02T09:19:02.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2023-11-02T09:19:02.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2023-11-02T09:19:02.693 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:19:02.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2023-11-02T09:19:02.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2023-11-02T09:19:03.109 INFO:tasks.workunit.client.0.smithi116.stderr:marked down osd.0. 2023-11-02T09:19:03.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2023-11-02T09:19:13.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2023-11-02T09:19:13.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2023-11-02T09:19:13.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2023-11-02T09:19:13.637 INFO:tasks.workunit.client.0.smithi116.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2023-11-02T09:19:13.638 INFO:tasks.workunit.client.0.smithi116.stdout:-1 0.29306 root default 2023-11-02T09:19:13.638 INFO:tasks.workunit.client.0.smithi116.stdout:-3 0.29306 host smithi116 2023-11-02T09:19:13.638 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:19:13.639 INFO:tasks.workunit.client.0.smithi116.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2023-11-02T09:19:13.639 INFO:tasks.workunit.client.0.smithi116.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2023-11-02T09:19:13.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2023-11-02T09:19:13.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2023-11-02T09:19:13.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2023-11-02T09:19:14.062 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:19:14.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2023-11-02T09:19:14.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2023-11-02T09:19:14.477 INFO:tasks.workunit.client.0.smithi116.stderr:marked down osd.0. 2023-11-02T09:19:14.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2023-11-02T09:19:24.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: ceph osd tree 2023-11-02T09:19:24.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep down 2023-11-02T09:19:24.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep osd.0 2023-11-02T09:19:24.910 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2023-11-02T09:19:24.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2023-11-02T09:19:24.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2023-11-02T09:19:24.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:19:24.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2023-11-02T09:19:24.912 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:19:24.912 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:19:24.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:19:24.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:19:24.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:19:25.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:19:25.018 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:19:25.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:19:25.019 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:19:25.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:19:25.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:19:25.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:19:25.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:19:25.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:19:25.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:19:25.021 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:19:25.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:19:25.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:19:25.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2023-11-02T09:19:25.036 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:19:25.037 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:19:25.037 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:19:25.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.235308 2023-11-02T09:19:25.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:19:25.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:19:25.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2023-11-02T09:19:25.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2023-11-02T09:19:25.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2023-11-02T09:19:25.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2023-11-02T09:19:25.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2023-11-02T09:19:25.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:19:25.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2023-11-02T09:19:25.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:19:25.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:19:25.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:19:25.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:19:25.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:19:25.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:19:25.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:19:25.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:19:25.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:19:25.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:19:25.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:19:25.044 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:19:25.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:19:25.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:19:25.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:19:25.045 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:19:25.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:19:25.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:19:25.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2023-11-02T09:19:25.046 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:19:25.047 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:19:25.047 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:19:25.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.235308 2023-11-02T09:19:25.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:19:25.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:19:25.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2023-11-02T09:19:25.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T09:19:25.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:19:25.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:19:25.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.235308 2023-11-02T09:19:25.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T09:19:25.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-11-02T09:19:25.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T09:19:25.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2023-11-02T09:19:25.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_osd_stop td/osd-markdown 2023-11-02T09:19:25.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:132: TEST_osd_stop: local dir=td/osd-markdown 2023-11-02T09:19:25.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:134: TEST_osd_stop: run_mon td/osd-markdown a 2023-11-02T09:19:25.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2023-11-02T09:19:25.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T09:19:25.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T09:19:25.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T09:19:25.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2023-11-02T09:19:25.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2023-11-02T09:19:25.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T09:19:25.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:19:25.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:19:25.456 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:19:25.456 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:19:25.456 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:19:25.456 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:19:25.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T09:19:25.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T09:19:25.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T09:19:25.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:19:25.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:19:25.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T09:19:25.625 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T09:19:25.625 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:19:25.625 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:19:25.625 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:19:25.625 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:19:25.626 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:19:25.626 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:19:25.626 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.235308/ceph-mon.a.asok 2023-11-02T09:19:25.626 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:19:25.626 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.235308/ceph-mon.a.asok config get fsid 2023-11-02T09:19:25.706 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T09:19:25.706 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:19:25.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:19:25.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T09:19:25.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T09:19:25.707 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:19:25.707 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:19:25.708 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:19:25.708 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:19:25.708 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:19:25.708 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:19:25.708 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.235308/ceph-mon.a.asok 2023-11-02T09:19:25.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:19:25.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.235308/ceph-mon.a.asok config get mon_host 2023-11-02T09:19:25.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:135: TEST_osd_stop: run_mgr td/osd-markdown x 2023-11-02T09:19:25.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2023-11-02T09:19:25.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T09:19:25.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T09:19:25.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T09:19:25.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2023-11-02T09:19:25.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T09:19:26.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T09:19:26.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:19:26.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:19:26.627 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:19:26.628 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:19:26.628 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:19:26.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:19:26.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:19:26.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:19:26.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:136: TEST_osd_stop: run_osd td/osd-markdown 0 2023-11-02T09:19:26.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2023-11-02T09:19:26.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:19:26.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T09:19:26.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:19:26.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2023-11-02T09:19:26.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 ' 2023-11-02T09:19:26.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:19:26.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:19:26.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:19:26.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2023-11-02T09:19:26.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2023-11-02T09:19:26.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:19:26.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:19:26.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2023-11-02T09:19:26.658 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:19:26.658 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:19:26.658 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:19:26.658 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:19:26.659 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:19:26.659 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:19:26.659 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:19:26.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:19:26.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:19:26.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:19:26.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:19:26.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2023-11-02T09:19:26.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2023-11-02T09:19:26.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:19:26.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:19:26.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:19:26.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:19:26.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:19:26.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:19:26.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2023-11-02T09:19:26.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:19:26.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8fa5b6a0-d8b6-4bf5-9942-6e61c35f74e7 2023-11-02T09:19:26.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8fa5b6a0-d8b6-4bf5-9942-6e61c35f74e7' 2023-11-02T09:19:26.663 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 8fa5b6a0-d8b6-4bf5-9942-6e61c35f74e7 2023-11-02T09:19:26.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:19:26.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCeaUNl8GzZJxAAjyTd+HZwK1Eyvq30lkvTAg== 2023-11-02T09:19:26.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCeaUNl8GzZJxAAjyTd+HZwK1Eyvq30lkvTAg=="}' 2023-11-02T09:19:26.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8fa5b6a0-d8b6-4bf5-9942-6e61c35f74e7 -i td/osd-markdown/0/new.json 2023-11-02T09:19:27.006 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:19:27.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2023-11-02T09:19:27.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCeaUNl8GzZJxAAjyTd+HZwK1Eyvq30lkvTAg== --osd-uuid 8fa5b6a0-d8b6-4bf5-9942-6e61c35f74e7 2023-11-02T09:19:27.042 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:19:27.039+0000 7f9e5d025500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:19:27.043 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:19:27.041+0000 7f9e5d025500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:19:27.044 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:19:27.042+0000 7f9e5d025500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:19:27.045 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:19:27.042+0000 7f9e5d025500 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2023-11-02T09:19:27.045 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:19:27.042+0000 7f9e5d025500 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2023-11-02T09:19:27.046 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:19:27.042+0000 7f9e5d025500 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2023-11-02T09:19:27.097 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:19:27.094+0000 7f9e5d025500 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2023-11-02T09:19:32.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2023-11-02T09:19:32.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:19:32.415 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T09:19:32.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T09:19:32.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:19:32.928 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T09:19:32.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T09:19:32.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:19:32.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:19:32.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:19:32.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:19:32.953 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:19:32.951+0000 7f9c37425500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:19:32.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:19:32.959+0000 7f9c37425500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:19:32.963 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:19:32.960+0000 7f9c37425500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:19:33.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T09:19:33.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:19:33.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T09:19:33.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:19:33.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:19:33.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:19:33.334 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:19:33.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:19:33.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:19:33.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:19:33.497 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:19:33.495+0000 7f9c37425500 -1 Falling back to public interface 2023-11-02T09:19:33.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:19:34.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:19:34.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:19:34.735 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:19:34.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:19:34.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:19:34.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:19:35.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:19:35.838 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:19:35.835+0000 7f9c37425500 -1 osd.0 0 log_to_monitors true 2023-11-02T09:19:36.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:19:36.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:19:36.153 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:19:36.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:19:36.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:19:36.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:19:36.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:19:37.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:19:37.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:19:37.627 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:19:37.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:19:37.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:19:37.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:19:38.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:19:39.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:19:39.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:19:39.012 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:19:39.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:19:39.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:19:39.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:19:39.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:19:40.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:19:40.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:19:40.409 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:19:40.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:19:40.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:19:40.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:19:40.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:19:41.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:19:41.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:19:41.914 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:19:41.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:19:41.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:19:41.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:19:42.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:19:43.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:19:43.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:19:43.295 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:19:43.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:19:43.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:19:43.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:19:43.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:19:44.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:19:44.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:19:44.720 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:19:44.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:19:44.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:19:44.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:19:45.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:19:46.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:19:46.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:19:46.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:19:46.102 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:19:46.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:19:46.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:19:46.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:19:47.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:19:47.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:19:47.605 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:19:47.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:19:47.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:19:47.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:19:48.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:19:49.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:19:49.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:19:49.342 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:19:49.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:19:49.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:19:49.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:19:49.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:19:50.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:19:50.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:19:50.890 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:19:50.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:19:50.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:19:50.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:19:51.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:19:52.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:19:52.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:19:52.874 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:19:52.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:19:52.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:19:52.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:19:53.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:19:54.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:19:54.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:19:54.891 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:19:54.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:19:54.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:19:54.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:19:55.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:19:56.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:19:56.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:19:56.971 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:19:56.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:19:56.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:19:56.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:19:57.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:19:58.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:19:58.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:19:58.869 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:19:58.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:19:58.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:19:58.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:19:59.285 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/254169,v1:127.0.0.1:6803/254169] [v2:127.0.0.1:6804/254169,v1:127.0.0.1:6805/254169] exists,up 8fa5b6a0-d8b6-4bf5-9942-6e61c35f74e7 2023-11-02T09:19:59.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:19:59.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:19:59.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:19:59.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:137: TEST_osd_stop: run_osd td/osd-markdown 1 2023-11-02T09:19:59.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2023-11-02T09:19:59.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:19:59.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-11-02T09:19:59.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:19:59.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2023-11-02T09:19:59.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 ' 2023-11-02T09:19:59.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:19:59.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:19:59.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:19:59.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2023-11-02T09:19:59.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2023-11-02T09:19:59.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:19:59.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:19:59.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2023-11-02T09:19:59.296 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:19:59.296 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:19:59.296 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:19:59.297 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:19:59.297 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:19:59.297 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:19:59.297 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:19:59.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:19:59.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:19:59.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:19:59.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:19:59.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2023-11-02T09:19:59.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2023-11-02T09:19:59.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:19:59.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:19:59.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:19:59.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:19:59.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:19:59.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:19:59.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2023-11-02T09:19:59.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:19:59.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=11f8bbe6-5b37-45ca-8c40-697c82917ba7 2023-11-02T09:19:59.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 11f8bbe6-5b37-45ca-8c40-697c82917ba7' 2023-11-02T09:19:59.301 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 11f8bbe6-5b37-45ca-8c40-697c82917ba7 2023-11-02T09:19:59.301 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:19:59.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC/aUNliJD8ERAAhYBtfLdhlpFZK1t5uE8iEg== 2023-11-02T09:19:59.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC/aUNliJD8ERAAhYBtfLdhlpFZK1t5uE8iEg=="}' 2023-11-02T09:19:59.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 11f8bbe6-5b37-45ca-8c40-697c82917ba7 -i td/osd-markdown/1/new.json 2023-11-02T09:19:59.793 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:19:59.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2023-11-02T09:19:59.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC/aUNliJD8ERAAhYBtfLdhlpFZK1t5uE8iEg== --osd-uuid 11f8bbe6-5b37-45ca-8c40-697c82917ba7 2023-11-02T09:19:59.831 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:19:59.827+0000 7f4784aa5500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:19:59.832 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:19:59.829+0000 7f4784aa5500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:19:59.833 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:19:59.830+0000 7f4784aa5500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:19:59.833 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:19:59.830+0000 7f4784aa5500 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2023-11-02T09:19:59.834 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:19:59.830+0000 7f4784aa5500 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2023-11-02T09:19:59.834 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:19:59.830+0000 7f4784aa5500 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2023-11-02T09:19:59.908 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:19:59.905+0000 7f4784aa5500 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2023-11-02T09:20:05.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2023-11-02T09:20:05.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:20:05.722 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2023-11-02T09:20:05.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-11-02T09:20:05.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:20:06.235 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T09:20:06.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-11-02T09:20:06.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:20:06.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:20:06.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:20:06.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:20:06.259 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:20:06.256+0000 7efe6ac15500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:20:06.279 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:20:06.276+0000 7efe6ac15500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:20:06.280 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:20:06.277+0000 7efe6ac15500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:20:06.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-11-02T09:20:06.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:20:06.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T09:20:06.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:20:06.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:20:06.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:06.683 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:20:06.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:20:06.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:06.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:20:06.812 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:20:06.810+0000 7efe6ac15500 -1 Falling back to public interface 2023-11-02T09:20:07.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:08.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:08.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:08.055 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:20:08.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:20:08.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:08.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:20:08.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:09.242 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:20:09.239+0000 7efe6ac15500 -1 osd.1 0 log_to_monitors true 2023-11-02T09:20:09.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:09.464 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:20:09.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:09.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:20:09.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:09.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:20:09.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:10.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:10.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:10.918 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:20:10.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:20:10.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:10.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:20:11.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:12.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:12.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:12.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:20:12.309 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:20:12.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:12.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:20:12.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:13.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:13.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:13.689 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:20:13.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:20:13.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:13.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:20:14.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:15.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:15.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:15.158 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:20:15.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:20:15.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:15.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:20:15.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:16.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:16.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:16.576 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:20:16.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:20:16.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:16.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:20:16.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:17.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:18.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:18.000 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:20:18.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:20:18.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:18.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:20:18.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:19.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:19.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:19.392 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:20:19.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:20:19.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:19.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:20:19.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:20.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:20.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:20.794 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:20:20.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:20:20.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:20.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:20:21.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:22.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:22.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:22.183 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:20:22.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:20:22.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:22.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:20:23.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:24.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:24.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:24.399 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:20:24.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:20:24.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:24.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:20:25.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:26.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:26.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:26.352 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:20:26.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:20:26.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:26.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:20:27.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:28.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:28.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:28.104 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:20:28.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:20:28.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:28.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:20:28.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:29.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:29.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:29.721 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:20:29.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:20:29.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:29.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:20:30.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:31.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:31.530 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:20:31.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:31.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:20:31.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:31.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:20:31.902 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/255882,v1:127.0.0.1:6811/255882] [v2:127.0.0.1:6812/255882,v1:127.0.0.1:6813/255882] exists,up 11f8bbe6-5b37-45ca-8c40-697c82917ba7 2023-11-02T09:20:31.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:20:31.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:20:31.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:20:31.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:138: TEST_osd_stop: run_osd td/osd-markdown 2 2023-11-02T09:20:31.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2023-11-02T09:20:31.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:20:31.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-11-02T09:20:31.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:20:31.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2023-11-02T09:20:31.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 ' 2023-11-02T09:20:31.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:20:31.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:20:31.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:20:31.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2023-11-02T09:20:31.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2023-11-02T09:20:31.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:20:31.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:20:31.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2023-11-02T09:20:31.911 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:20:31.911 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:20:31.911 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:20:31.911 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:20:31.911 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:20:31.912 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:20:31.912 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:20:31.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' 2023-11-02T09:20:31.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:20:31.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:20:31.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:20:31.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2023-11-02T09:20:31.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2023-11-02T09:20:31.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:20:31.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:20:31.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:20:31.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:20:31.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:20:31.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:20:31.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2023-11-02T09:20:31.914 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:20:31.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6f69862a-5dbe-4331-90c9-e7af836483c0 2023-11-02T09:20:31.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6f69862a-5dbe-4331-90c9-e7af836483c0' 2023-11-02T09:20:31.915 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 6f69862a-5dbe-4331-90c9-e7af836483c0 2023-11-02T09:20:31.916 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:20:31.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDfaUNlFsq+NhAAB9WOzJJyMqa0XxqEsKvfZw== 2023-11-02T09:20:31.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDfaUNlFsq+NhAAB9WOzJJyMqa0XxqEsKvfZw=="}' 2023-11-02T09:20:31.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6f69862a-5dbe-4331-90c9-e7af836483c0 -i td/osd-markdown/2/new.json 2023-11-02T09:20:32.466 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:20:32.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2023-11-02T09:20:32.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDfaUNlFsq+NhAAB9WOzJJyMqa0XxqEsKvfZw== --osd-uuid 6f69862a-5dbe-4331-90c9-e7af836483c0 2023-11-02T09:20:32.507 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:20:32.500+0000 7f4c2438b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:20:32.516 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:20:32.513+0000 7f4c2438b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:20:32.517 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:20:32.514+0000 7f4c2438b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:20:32.517 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:20:32.515+0000 7f4c2438b500 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2023-11-02T09:20:32.518 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:20:32.515+0000 7f4c2438b500 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2023-11-02T09:20:32.518 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:20:32.515+0000 7f4c2438b500 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2023-11-02T09:20:32.591 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:20:32.588+0000 7f4c2438b500 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2023-11-02T09:20:38.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2023-11-02T09:20:38.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:20:38.197 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2023-11-02T09:20:38.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-11-02T09:20:38.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:20:38.709 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T09:20:38.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-11-02T09:20:38.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=eae3aac6-d550-41cf-9722-e18d1098e64f --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.235308/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:20:38.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:20:38.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:20:38.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:20:38.733 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:20:38.730+0000 7eff114da500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:20:38.747 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:20:38.744+0000 7eff114da500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:20:38.748 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:20:38.745+0000 7eff114da500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:20:39.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-11-02T09:20:39.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:20:39.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T09:20:39.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:20:39.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:20:39.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:39.145 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:20:39.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:20:39.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:39.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:20:39.288 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:20:39.285+0000 7eff114da500 -1 Falling back to public interface 2023-11-02T09:20:39.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:40.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:40.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:40.555 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:20:40.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:20:40.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:40.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:20:41.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:41.566 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:20:41.564+0000 7eff114da500 -1 osd.2 0 log_to_monitors true 2023-11-02T09:20:42.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:42.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:42.008 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:20:42.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:20:42.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:42.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:20:42.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:43.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:43.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:43.598 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:20:43.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:20:43.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:43.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:20:44.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:45.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:45.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:45.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:20:45.031 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:20:45.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:45.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:20:45.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:46.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:46.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:46.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:20:46.519 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:20:46.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:46.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:20:46.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:48.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:48.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:48.000 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:20:48.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:20:48.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:48.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:20:48.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:49.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:49.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:49.447 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:20:49.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:20:49.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:49.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:20:49.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:50.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:50.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:50.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:20:50.828 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:20:50.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:50.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:20:51.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:52.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:52.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:52.316 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:20:52.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:20:52.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:52.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:20:52.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:53.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:53.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:53.742 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:20:53.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:20:53.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:53.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:20:54.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:55.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:55.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:55.313 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:20:55.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:20:55.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:55.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:20:56.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:57.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:57.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:57.035 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:20:57.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:20:57.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:57.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:20:57.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:20:58.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:20:58.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:20:58.552 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:20:58.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:20:58.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:20:58.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:20:59.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:00.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:00.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:00.085 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:21:00.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:21:00.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:00.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:21:00.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:01.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:01.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:01.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:21:01.704 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:21:01.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:01.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:21:02.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:03.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:03.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:03.486 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:21:03.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:21:03.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:03.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:21:04.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:05.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:05.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:05.154 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T09:21:05.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T09:21:05.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:05.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:21:05.592 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/257597,v1:127.0.0.1:6819/257597] [v2:127.0.0.1:6820/257597,v1:127.0.0.1:6821/257597] exists,up 6f69862a-5dbe-4331-90c9-e7af836483c0 2023-11-02T09:21:05.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:21:05.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:21:05.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:21:05.593 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: cat td/osd-markdown/osd.0.pid 2023-11-02T09:21:05.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: osd_0_pid=254169 2023-11-02T09:21:05.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:140: TEST_osd_stop: ps -p 254169 2023-11-02T09:21:05.601 INFO:tasks.workunit.client.0.smithi116.stdout: PID TTY TIME CMD 2023-11-02T09:21:05.601 INFO:tasks.workunit.client.0.smithi116.stdout: 254169 ? 00:00:00 ceph-osd 2023-11-02T09:21:05.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: ceph osd tree 2023-11-02T09:21:05.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep osd.0 2023-11-02T09:21:05.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep up 2023-11-02T09:21:05.985 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2023-11-02T09:21:05.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:143: TEST_osd_stop: ceph osd stop osd.0 2023-11-02T09:21:06.511 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 is already stopped. 2023-11-02T09:21:06.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:144: TEST_osd_stop: sleep 15 2023-11-02T09:21:21.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: ceph osd tree 2023-11-02T09:21:21.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep down 2023-11-02T09:21:21.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep osd.0 2023-11-02T09:21:21.900 INFO:tasks.workunit.client.0.smithi116.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2023-11-02T09:21:21.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:146: TEST_osd_stop: ps -p 254169 2023-11-02T09:21:21.907 INFO:tasks.workunit.client.0.smithi116.stdout: PID TTY TIME CMD 2023-11-02T09:21:21.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2023-11-02T09:21:21.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2023-11-02T09:21:21.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:21:21.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2023-11-02T09:21:21.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:21:21.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:21:21.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:21:21.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:21:21.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:21:22.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:21:22.015 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:21:22.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:21:22.015 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:21:22.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:21:22.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:21:22.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:21:22.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:21:22.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:21:22.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:21:22.019 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:21:22.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:21:22.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:21:22.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2023-11-02T09:21:22.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:21:22.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:21:22.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:21:22.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.235308 2023-11-02T09:21:22.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:21:22.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:21:22.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2023-11-02T09:21:22.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-markdown 0 2023-11-02T09:21:22.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2023-11-02T09:21:22.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-11-02T09:21:22.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2023-11-02T09:21:22.035 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:21:22.035 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:21:22.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:21:22.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:21:22.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:21:22.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:21:22.037 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:21:22.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:21:22.037 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:21:22.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:21:22.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:21:22.038 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:21:22.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:21:22.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:21:22.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:21:22.040 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:21:22.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:21:22.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-11-02T09:21:22.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2023-11-02T09:21:22.042 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:21:22.042 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:21:22.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.235308 2023-11-02T09:21:22.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.235308 2023-11-02T09:21:22.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:21:22.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:21:22.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2023-11-02T09:21:22.044 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-02T09:21:22.044 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-02T09:21:22.110 INFO:tasks.workunit:Running workunit osd/osd-reactivate.sh... 2023-11-02T09:21:22.110 DEBUG:teuthology.orchestra.run.smithi116:workunit test osd/osd-reactivate.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh 2023-11-02T09:21:22.171 INFO:tasks.workunit.client.0.smithi116.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-11-02T09:21:22.177 INFO:tasks.workunit.client.0.smithi116.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-11-02T09:21:22.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T09:21:22.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T09:21:22.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2023-11-02T09:21:22.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2023-11-02T09:21:22.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2023-11-02T09:21:22.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2023-11-02T09:21:22.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2023-11-02T09:21:22.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2023-11-02T09:21:22.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-reactivate 2023-11-02T09:21:22.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:19: run: local dir=td/osd-reactivate 2023-11-02T09:21:22.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:20: run: shift 2023-11-02T09:21:22.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: export CEPH_MON=127.0.0.1:7122 2023-11-02T09:21:22.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: CEPH_MON=127.0.0.1:7122 2023-11-02T09:21:22.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:23: run: export CEPH_ARGS 2023-11-02T09:21:22.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: uuidgen 2023-11-02T09:21:22.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: CEPH_ARGS+='--fsid=a2ea7144-5432-4bda-87d9-9c0d0b999253 --auth-supported=none ' 2023-11-02T09:21:22.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:25: run: CEPH_ARGS+='--mon-host=127.0.0.1:7122 ' 2023-11-02T09:21:22.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: set 2023-11-02T09:21:22.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-11-02T09:21:22.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: local funcs=TEST_reactivate 2023-11-02T09:21:22.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:28: run: for func in $funcs 2023-11-02T09:21:22.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:29: run: setup td/osd-reactivate 2023-11-02T09:21:22.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reactivate 2023-11-02T09:21:22.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reactivate 2023-11-02T09:21:22.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2023-11-02T09:21:22.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:21:22.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2023-11-02T09:21:22.189 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:21:22.189 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:21:22.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:21:22.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:21:22.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:21:22.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:21:22.190 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:21:22.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:21:22.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:21:22.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:21:22.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:21:22.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:21:22.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:21:22.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:21:22.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:21:22.192 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:21:22.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:21:22.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:21:22.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2023-11-02T09:21:22.193 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:21:22.193 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:21:22.193 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258663 2023-11-02T09:21:22.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.258663 2023-11-02T09:21:22.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:21:22.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:21:22.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reactivate 2023-11-02T09:21:22.194 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T09:21:22.194 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:21:22.195 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258663 2023-11-02T09:21:22.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.258663 2023-11-02T09:21:22.196 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T09:21:22.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-11-02T09:21:22.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-11-02T09:21:22.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T09:21:22.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reactivate 1' TERM HUP INT 2023-11-02T09:21:22.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:30: run: TEST_reactivate td/osd-reactivate 2023-11-02T09:21:22.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:36: TEST_reactivate: local dir=td/osd-reactivate 2023-11-02T09:21:22.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:38: TEST_reactivate: run_mon td/osd-reactivate a 2023-11-02T09:21:22.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reactivate 2023-11-02T09:21:22.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T09:21:22.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T09:21:22.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T09:21:22.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reactivate/a 2023-11-02T09:21:22.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reactivate/a --run-dir=td/osd-reactivate 2023-11-02T09:21:22.518 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T09:21:22.519 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:21:22.519 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:21:22.519 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:21:22.519 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:21:22.520 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258663 2023-11-02T09:21:22.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.258663/$cluster-$name.asok' 2023-11-02T09:21:22.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reactivate/a '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.258663/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reactivate/log --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T09:21:22.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T09:21:22.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T09:21:22.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:21:22.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:21:22.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T09:21:22.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T09:21:22.711 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:21:22.711 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:21:22.711 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:21:22.711 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:21:22.711 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:21:22.712 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258663 2023-11-02T09:21:22.712 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.258663/ceph-mon.a.asok 2023-11-02T09:21:22.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:21:22.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.258663/ceph-mon.a.asok config get fsid 2023-11-02T09:21:22.790 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T09:21:22.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:21:22.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:21:22.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T09:21:22.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T09:21:22.792 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:21:22.792 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:21:22.792 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:21:22.792 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:21:22.792 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:21:22.793 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258663 2023-11-02T09:21:22.793 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.258663/ceph-mon.a.asok 2023-11-02T09:21:22.793 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:21:22.793 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.258663/ceph-mon.a.asok config get mon_host 2023-11-02T09:21:22.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:39: TEST_reactivate: run_mgr td/osd-reactivate x 2023-11-02T09:21:22.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-reactivate 2023-11-02T09:21:22.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T09:21:22.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T09:21:22.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T09:21:22.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-reactivate/x 2023-11-02T09:21:22.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T09:21:23.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T09:21:23.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:21:23.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:21:23.716 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:21:23.716 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:21:23.717 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258663 2023-11-02T09:21:23.717 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.258663/$cluster-$name.asok' 2023-11-02T09:21:23.717 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:21:23.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reactivate/x '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.258663/$cluster-$name.asok' --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:21:23.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:40: TEST_reactivate: run_osd td/osd-reactivate 0 2023-11-02T09:21:23.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reactivate 2023-11-02T09:21:23.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:21:23.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T09:21:23.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:21:23.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reactivate/0 2023-11-02T09:21:23.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a2ea7144-5432-4bda-87d9-9c0d0b999253 --auth-supported=none --mon-host=127.0.0.1:7122 ' 2023-11-02T09:21:23.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:21:23.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:21:23.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:21:23.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2023-11-02T09:21:23.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2023-11-02T09:21:23.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:21:23.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:21:23.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2023-11-02T09:21:23.749 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:21:23.749 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:21:23.749 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:21:23.750 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:21:23.750 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:21:23.750 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258663 2023-11-02T09:21:23.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.258663/$cluster-$name.asok' 2023-11-02T09:21:23.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.258663/$cluster-$name.asok' 2023-11-02T09:21:23.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:21:23.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:21:23.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:21:23.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2023-11-02T09:21:23.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2023-11-02T09:21:23.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:21:23.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:21:23.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:21:23.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:21:23.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:21:23.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:21:23.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reactivate/0 2023-11-02T09:21:23.753 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:21:23.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ce8aa9fe-9b7b-4f68-9835-1557b05a04d2 2023-11-02T09:21:23.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ce8aa9fe-9b7b-4f68-9835-1557b05a04d2' 2023-11-02T09:21:23.754 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 ce8aa9fe-9b7b-4f68-9835-1557b05a04d2 2023-11-02T09:21:23.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:21:23.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQATakNlmOkvLRAAyLN96VXrupTwtkHZxIl/fQ== 2023-11-02T09:21:23.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQATakNlmOkvLRAAyLN96VXrupTwtkHZxIl/fQ=="}' 2023-11-02T09:21:23.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ce8aa9fe-9b7b-4f68-9835-1557b05a04d2 -i td/osd-reactivate/0/new.json 2023-11-02T09:21:24.109 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:21:24.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reactivate/0/new.json 2023-11-02T09:21:24.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=a2ea7144-5432-4bda-87d9-9c0d0b999253 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.258663/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQATakNlmOkvLRAAyLN96VXrupTwtkHZxIl/fQ== --osd-uuid ce8aa9fe-9b7b-4f68-9835-1557b05a04d2 2023-11-02T09:21:24.145 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:21:24.142+0000 7f698ff1a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:21:24.147 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:21:24.145+0000 7f698ff1a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:21:24.148 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:21:24.146+0000 7f698ff1a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:21:24.149 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:21:24.146+0000 7f698ff1a500 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2023-11-02T09:21:24.149 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:21:24.146+0000 7f698ff1a500 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2023-11-02T09:21:24.149 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:21:24.146+0000 7f698ff1a500 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2023-11-02T09:21:24.233 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:21:24.230+0000 7f698ff1a500 -1 bluestore(td/osd-reactivate/0) _read_fsid unparsable uuid 2023-11-02T09:21:29.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reactivate/0/keyring 2023-11-02T09:21:29.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:21:29.507 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T09:21:29.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T09:21:29.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reactivate/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:21:30.054 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T09:21:30.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T09:21:30.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=a2ea7144-5432-4bda-87d9-9c0d0b999253 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.258663/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:21:30.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:21:30.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:21:30.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:21:30.078 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:21:30.075+0000 7f67e0e10500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:21:30.080 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:21:30.077+0000 7f67e0e10500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:21:30.081 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:21:30.078+0000 7f67e0e10500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:21:30.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T09:21:30.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:21:30.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T09:21:30.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:21:30.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:21:30.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:30.497 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:21:30.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:21:30.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:30.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:21:30.613 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:21:30.606+0000 7f67e0e10500 -1 Falling back to public interface 2023-11-02T09:21:30.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:31.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:31.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:31.905 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:21:31.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:21:31.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:31.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:21:32.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:32.907 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:21:32.904+0000 7f67e0e10500 -1 osd.0 0 log_to_monitors true 2023-11-02T09:21:33.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:33.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:33.323 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:21:33.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:21:33.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:33.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:21:33.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:34.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:34.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:34.702 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:21:34.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:21:34.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:34.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:21:35.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:36.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:36.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:36.100 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:21:36.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:21:36.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:36.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:21:36.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:37.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:37.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:37.484 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:21:37.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:21:37.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:37.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:21:37.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:38.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:38.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:38.939 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:21:38.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:21:38.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:38.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:21:39.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:40.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:40.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:40.317 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:21:40.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:21:40.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:40.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:21:40.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:41.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:41.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:41.795 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:21:41.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:21:41.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:41.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:21:42.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:43.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:43.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:43.192 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:21:43.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:21:43.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:43.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:21:43.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:44.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:44.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:44.657 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:21:44.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:21:44.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:44.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:21:45.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:46.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:46.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:46.410 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:21:46.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:21:46.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:46.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:21:46.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:47.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:47.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:47.920 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:21:47.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:21:47.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:47.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:21:48.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:49.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:49.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:49.478 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:21:49.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:21:49.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:49.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:21:49.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:50.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:50.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:50.987 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:21:50.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:21:50.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:50.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:21:52.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:53.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:53.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:53.137 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:21:53.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:21:53.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:53.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:21:53.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:54.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:54.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:54.919 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:21:54.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:21:54.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:54.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:21:55.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:56.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:56.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:56.383 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T09:21:56.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T09:21:56.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:56.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:21:56.797 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/259759,v1:127.0.0.1:6803/259759] [v2:127.0.0.1:6804/259759,v1:127.0.0.1:6805/259759] exists,up ce8aa9fe-9b7b-4f68-9835-1557b05a04d2 2023-11-02T09:21:56.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:21:56.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:21:56.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:21:56.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:42: TEST_reactivate: kill_daemons td/osd-reactivate TERM osd 2023-11-02T09:21:56.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:21:56.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:21:56.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:21:56.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:21:56.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:21:57.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:21:57.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:44: TEST_reactivate: ready_path=td/osd-reactivate/0/ready 2023-11-02T09:21:57.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:45: TEST_reactivate: activate_path=td/osd-reactivate/0/active 2023-11-02T09:21:57.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:47: TEST_reactivate: rm -rf td/osd-reactivate/0/ready td/osd-reactivate/0/active 2023-11-02T09:21:57.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:48: TEST_reactivate: activate_osd td/osd-reactivate 0 2023-11-02T09:21:57.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-reactivate 2023-11-02T09:21:57.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-11-02T09:21:57.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-11-02T09:21:57.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-11-02T09:21:57.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-reactivate/0 2023-11-02T09:21:57.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=a2ea7144-5432-4bda-87d9-9c0d0b999253 --auth-supported=none --mon-host=127.0.0.1:7122 ' 2023-11-02T09:21:57.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:21:57.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:21:57.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:21:57.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2023-11-02T09:21:57.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2023-11-02T09:21:57.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-11-02T09:21:57.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-11-02T09:21:57.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2023-11-02T09:21:57.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-11-02T09:21:57.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:21:57.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:21:57.111 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:21:57.111 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:21:57.112 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258663 2023-11-02T09:21:57.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.258663/$cluster-$name.asok' 2023-11-02T09:21:57.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.258663/$cluster-$name.asok' 2023-11-02T09:21:57.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:21:57.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2023-11-02T09:21:57.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2023-11-02T09:21:57.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:21:57.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:21:57.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:21:57.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:21:57.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-11-02T09:21:57.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-11-02T09:21:57.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-reactivate/0 2023-11-02T09:21:57.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-11-02T09:21:57.115 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T09:21:57.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=a2ea7144-5432-4bda-87d9-9c0d0b999253 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.258663/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:21:57.116 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-reactivate/0/whoami 2023-11-02T09:21:57.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-11-02T09:21:57.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-11-02T09:21:57.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-11-02T09:21:57.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-11-02T09:21:57.130 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:21:57.127+0000 7f1c71107500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:21:57.131 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:21:57.129+0000 7f1c71107500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:21:57.133 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:21:57.130+0000 7f1c71107500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:21:57.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-11-02T09:21:57.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:21:57.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T09:21:57.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:21:57.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:21:57.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:57.482 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:21:57.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:21:57.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:57.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:21:57.652 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:21:57.650+0000 7f1c71107500 -1 Falling back to public interface 2023-11-02T09:21:57.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:21:58.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:21:58.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:21:58.902 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:21:58.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:21:58.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:21:58.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:21:59.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:22:00.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:22:00.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:22:00.346 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:22:00.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:22:00.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:22:00.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:22:00.419 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:00.416+0000 7f1c71107500 -1 osd.0 6 log_to_monitors true 2023-11-02T09:22:00.773 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/260692,v1:127.0.0.1:6803/260692] [v2:127.0.0.1:6804/260692,v1:127.0.0.1:6805/260692] exists,up ce8aa9fe-9b7b-4f68-9835-1557b05a04d2 2023-11-02T09:22:00.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:22:00.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:22:00.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:22:00.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:31: run: teardown td/osd-reactivate 2023-11-02T09:22:00.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2023-11-02T09:22:00.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:22:00.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2023-11-02T09:22:00.776 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:22:00.776 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:22:00.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:22:00.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:22:00.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:22:00.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:22:00.880 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:22:00.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:22:00.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:22:00.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:22:00.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:22:00.882 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:22:00.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:22:00.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:22:00.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:22:00.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:22:00.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:22:00.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:22:00.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2023-11-02T09:22:00.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:22:00.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:22:00.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258663 2023-11-02T09:22:00.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.258663 2023-11-02T09:22:00.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:22:00.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:22:00.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2023-11-02T09:22:00.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-reactivate 0 2023-11-02T09:22:00.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2023-11-02T09:22:00.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-11-02T09:22:00.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2023-11-02T09:22:00.894 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:22:00.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:22:00.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:22:00.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:22:00.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:22:00.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:22:00.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:22:00.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:22:00.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:22:00.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:22:00.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:22:00.902 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:22:00.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:22:00.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:22:00.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:22:00.905 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:22:00.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:22:00.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-11-02T09:22:00.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2023-11-02T09:22:00.908 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:22:00.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:22:00.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258663 2023-11-02T09:22:00.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.258663 2023-11-02T09:22:00.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:22:00.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:22:00.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2023-11-02T09:22:00.910 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-02T09:22:00.910 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-02T09:22:00.976 INFO:tasks.workunit:Running workunit osd/osd-recovery-prio.sh... 2023-11-02T09:22:00.976 DEBUG:teuthology.orchestra.run.smithi116:workunit test osd/osd-recovery-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh 2023-11-02T09:22:01.036 INFO:tasks.workunit.client.0.smithi116.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-11-02T09:22:01.043 INFO:tasks.workunit.client.0.smithi116.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-11-02T09:22:01.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T09:22:01.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T09:22:01.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2023-11-02T09:22:01.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2023-11-02T09:22:01.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2023-11-02T09:22:01.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2023-11-02T09:22:01.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2023-11-02T09:22:01.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2023-11-02T09:22:01.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-recovery-prio 2023-11-02T09:22:01.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:21: run: local dir=td/osd-recovery-prio 2023-11-02T09:22:01.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:22: run: shift 2023-11-02T09:22:01.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2023-11-02T09:22:01.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2023-11-02T09:22:01.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:26: run: export CEPH_ARGS 2023-11-02T09:22:01.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: uuidgen 2023-11-02T09:22:01.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: CEPH_ARGS+='--fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none ' 2023-11-02T09:22:01.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2023-11-02T09:22:01.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:31: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2023-11-02T09:22:01.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: export objects=200 2023-11-02T09:22:01.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: objects=200 2023-11-02T09:22:01.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:33: run: export poolprefix=test 2023-11-02T09:22:01.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:33: run: poolprefix=test 2023-11-02T09:22:01.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: export FORCE_PRIO=255 2023-11-02T09:22:01.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: FORCE_PRIO=255 2023-11-02T09:22:01.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: export NORMAL_PRIO=190 2023-11-02T09:22:01.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: NORMAL_PRIO=190 2023-11-02T09:22:01.054 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: set 2023-11-02T09:22:01.054 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-11-02T09:22:01.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: local 'funcs=TEST_recovery_pool_priority 2023-11-02T09:22:01.060 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_recovery_priority' 2023-11-02T09:22:01.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: for func in $funcs 2023-11-02T09:22:01.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:39: run: setup td/osd-recovery-prio 2023-11-02T09:22:01.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2023-11-02T09:22:01.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2023-11-02T09:22:01.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2023-11-02T09:22:01.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:22:01.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2023-11-02T09:22:01.062 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:22:01.062 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:22:01.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:22:01.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:22:01.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:22:01.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:22:01.063 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:22:01.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:22:01.063 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:22:01.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:22:01.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:22:01.064 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:22:01.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:22:01.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:22:01.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:22:01.065 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:22:01.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:22:01.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:22:01.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2023-11-02T09:22:01.066 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:22:01.066 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:22:01.066 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:22:01.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.261297 2023-11-02T09:22:01.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:22:01.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:22:01.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2023-11-02T09:22:01.067 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T09:22:01.067 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:22:01.067 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:22:01.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.261297 2023-11-02T09:22:01.068 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T09:22:01.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-11-02T09:22:01.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-11-02T09:22:01.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T09:22:01.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2023-11-02T09:22:01.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:40: run: TEST_recovery_pool_priority td/osd-recovery-prio 2023-11-02T09:22:01.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:326: TEST_recovery_pool_priority: local dir=td/osd-recovery-prio 2023-11-02T09:22:01.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:327: TEST_recovery_pool_priority: local pools=3 2023-11-02T09:22:01.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:328: TEST_recovery_pool_priority: local OSDS=2 2023-11-02T09:22:01.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:330: TEST_recovery_pool_priority: run_mon td/osd-recovery-prio a 2023-11-02T09:22:01.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2023-11-02T09:22:01.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T09:22:01.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T09:22:01.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T09:22:01.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2023-11-02T09:22:01.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2023-11-02T09:22:01.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T09:22:01.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:22:01.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:22:01.445 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:22:01.445 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:22:01.445 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:22:01.445 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.261297/$cluster-$name.asok' 2023-11-02T09:22:01.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T09:22:01.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T09:22:01.630 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T09:22:01.630 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:22:01.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:22:01.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T09:22:01.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T09:22:01.631 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:22:01.631 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:22:01.632 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:22:01.632 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:22:01.632 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:22:01.632 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:22:01.632 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.261297/ceph-mon.a.asok 2023-11-02T09:22:01.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:22:01.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.261297/ceph-mon.a.asok config get fsid 2023-11-02T09:22:01.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T09:22:01.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:22:01.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:22:01.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T09:22:01.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T09:22:01.708 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:22:01.709 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:22:01.709 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:22:01.709 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:22:01.709 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:22:01.710 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:22:01.710 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.261297/ceph-mon.a.asok 2023-11-02T09:22:01.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:22:01.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.261297/ceph-mon.a.asok config get mon_host 2023-11-02T09:22:01.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: run_mgr td/osd-recovery-prio x 2023-11-02T09:22:01.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2023-11-02T09:22:01.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T09:22:01.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T09:22:01.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T09:22:01.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2023-11-02T09:22:01.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T09:22:02.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T09:22:02.634 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:22:02.634 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:22:02.634 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:22:02.634 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:22:02.635 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:22:02.635 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.261297/$cluster-$name.asok' 2023-11-02T09:22:02.635 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:22:02.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:22:02.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:332: TEST_recovery_pool_priority: export CEPH_ARGS 2023-11-02T09:22:02.663 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: expr 2 - 1 2023-11-02T09:22:02.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: seq 0 1 2023-11-02T09:22:02.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-11-02T09:22:02.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 0 2023-11-02T09:22:02.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2023-11-02T09:22:02.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:22:02.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T09:22:02.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:22:02.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2023-11-02T09:22:02.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2023-11-02T09:22:02.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:22:02.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:22:02.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:22:02.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2023-11-02T09:22:02.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2023-11-02T09:22:02.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:22:02.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:22:02.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2023-11-02T09:22:02.670 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:22:02.671 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:22:02.671 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:22:02.671 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:22:02.671 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:22:02.671 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:22:02.672 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.261297/$cluster-$name.asok' 2023-11-02T09:22:02.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' 2023-11-02T09:22:02.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:22:02.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:22:02.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:22:02.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2023-11-02T09:22:02.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2023-11-02T09:22:02.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:22:02.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:22:02.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:22:02.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:22:02.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:22:02.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:22:02.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/0 2023-11-02T09:22:02.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:22:02.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8687a291-3e06-455b-8594-1dbf5a4e8581 2023-11-02T09:22:02.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8687a291-3e06-455b-8594-1dbf5a4e8581' 2023-11-02T09:22:02.676 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 8687a291-3e06-455b-8594-1dbf5a4e8581 2023-11-02T09:22:02.676 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:22:02.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA6akNlnYzAKBAAxIRZ+TeB8IaurmO9b6cevQ== 2023-11-02T09:22:02.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA6akNlnYzAKBAAxIRZ+TeB8IaurmO9b6cevQ=="}' 2023-11-02T09:22:02.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8687a291-3e06-455b-8594-1dbf5a4e8581 -i td/osd-recovery-prio/0/new.json 2023-11-02T09:22:03.045 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:22:03.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/0/new.json 2023-11-02T09:22:03.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA6akNlnYzAKBAAxIRZ+TeB8IaurmO9b6cevQ== --osd-uuid 8687a291-3e06-455b-8594-1dbf5a4e8581 2023-11-02T09:22:03.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:03.083+0000 7fa1c181a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:22:03.089 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:03.085+0000 7fa1c181a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:22:03.089 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:03.086+0000 7fa1c181a500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:22:03.090 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:03.086+0000 7fa1c181a500 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2023-11-02T09:22:03.090 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:03.086+0000 7fa1c181a500 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2023-11-02T09:22:03.091 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:03.086+0000 7fa1c181a500 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2023-11-02T09:22:03.153 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:03.149+0000 7fa1c181a500 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2023-11-02T09:22:08.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2023-11-02T09:22:08.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:22:08.469 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T09:22:08.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T09:22:08.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:22:08.983 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T09:22:08.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T09:22:08.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:22:08.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:22:08.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:22:08.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:22:09.011 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:09.008+0000 7fdd0ce0d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:22:09.012 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:09.010+0000 7fdd0ce0d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:22:09.013 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:09.011+0000 7fdd0ce0d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:22:09.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T09:22:09.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:22:09.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T09:22:09.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:22:09.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:22:09.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:22:09.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:22:09.354 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:22:09.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:22:09.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:22:09.535 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:09.532+0000 7fdd0ce0d500 -1 Falling back to public interface 2023-11-02T09:22:09.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:22:10.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:22:10.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:22:10.767 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:22:10.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:22:10.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:22:10.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:22:11.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:22:11.844 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:11.841+0000 7fdd0ce0d500 -1 osd.0 0 log_to_monitors true 2023-11-02T09:22:12.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:22:12.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:22:12.147 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:22:12.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:22:12.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:22:12.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:22:12.655 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/262383,v1:127.0.0.1:6803/262383] [v2:127.0.0.1:6804/262383,v1:127.0.0.1:6805/262383] exists,up 8687a291-3e06-455b-8594-1dbf5a4e8581 2023-11-02T09:22:12.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:22:12.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:22:12.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:22:12.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-11-02T09:22:12.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 1 2023-11-02T09:22:12.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2023-11-02T09:22:12.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:22:12.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-11-02T09:22:12.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:22:12.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2023-11-02T09:22:12.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2023-11-02T09:22:12.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:22:12.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:22:12.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:22:12.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2023-11-02T09:22:12.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2023-11-02T09:22:12.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:22:12.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:22:12.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2023-11-02T09:22:12.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:22:12.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:22:12.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:22:12.668 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:22:12.668 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:22:12.668 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:22:12.668 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.261297/$cluster-$name.asok' 2023-11-02T09:22:12.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' 2023-11-02T09:22:12.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:22:12.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:22:12.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:22:12.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2023-11-02T09:22:12.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2023-11-02T09:22:12.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:22:12.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:22:12.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:22:12.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:22:12.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:22:12.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:22:12.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/1 2023-11-02T09:22:12.672 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:22:12.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d5fe1e1a-89cd-44b4-acb9-fbda2d492ff9 2023-11-02T09:22:12.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d5fe1e1a-89cd-44b4-acb9-fbda2d492ff9' 2023-11-02T09:22:12.673 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 d5fe1e1a-89cd-44b4-acb9-fbda2d492ff9 2023-11-02T09:22:12.673 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:22:12.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBEakNllEsrKBAAMImY2+LrAK2kPh4NBu1eTg== 2023-11-02T09:22:12.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBEakNllEsrKBAAMImY2+LrAK2kPh4NBu1eTg=="}' 2023-11-02T09:22:12.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d5fe1e1a-89cd-44b4-acb9-fbda2d492ff9 -i td/osd-recovery-prio/1/new.json 2023-11-02T09:22:13.305 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:22:13.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/1/new.json 2023-11-02T09:22:13.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBEakNllEsrKBAAMImY2+LrAK2kPh4NBu1eTg== --osd-uuid d5fe1e1a-89cd-44b4-acb9-fbda2d492ff9 2023-11-02T09:22:13.346 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:13.343+0000 7f19ed60c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:22:13.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:13.361+0000 7f19ed60c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:22:13.364 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:13.362+0000 7f19ed60c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:22:13.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:13.362+0000 7f19ed60c500 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2023-11-02T09:22:13.365 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:13.362+0000 7f19ed60c500 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2023-11-02T09:22:13.366 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:13.362+0000 7f19ed60c500 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2023-11-02T09:22:13.439 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:13.436+0000 7f19ed60c500 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2023-11-02T09:22:18.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2023-11-02T09:22:18.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:22:18.885 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2023-11-02T09:22:18.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-11-02T09:22:18.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:22:19.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-11-02T09:22:19.397 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T09:22:19.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:22:19.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:22:19.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:22:19.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:22:19.424 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:19.422+0000 7f0c40d2b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:22:19.444 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:19.441+0000 7f0c40d2b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:22:19.445 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:19.442+0000 7f0c40d2b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:22:19.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-11-02T09:22:19.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:22:19.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T09:22:19.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:22:19.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:22:19.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:22:19.850 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:22:19.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:22:19.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:22:19.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:22:19.972 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:19.970+0000 7f0c40d2b500 -1 Falling back to public interface 2023-11-02T09:22:20.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:22:21.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:22:21.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:22:21.259 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:22:21.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:22:21.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:22:21.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:22:21.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:22:22.329 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:22:22.327+0000 7f0c40d2b500 -1 osd.1 0 log_to_monitors true 2023-11-02T09:22:22.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:22:22.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:22:22.678 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:22:22.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:22:22.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:22:22.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:22:23.082 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/263741,v1:127.0.0.1:6811/263741] [v2:127.0.0.1:6812/263741,v1:127.0.0.1:6813/263741] exists,up d5fe1e1a-89cd-44b4-acb9-fbda2d492ff9 2023-11-02T09:22:23.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:22:23.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:22:23.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:22:23.084 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: seq 1 3 2023-11-02T09:22:23.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2023-11-02T09:22:23.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test1 1 1 2023-11-02T09:22:23.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2023-11-02T09:22:23.678 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test1' already exists 2023-11-02T09:22:23.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:22:24.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2023-11-02T09:22:25.817 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 size to 2 2023-11-02T09:22:25.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2023-11-02T09:22:25.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test2 1 1 2023-11-02T09:22:25.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2023-11-02T09:22:26.529 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test2' already exists 2023-11-02T09:22:26.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:22:27.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2023-11-02T09:22:28.243 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 2 size to 2 2023-11-02T09:22:28.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2023-11-02T09:22:28.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test3 1 1 2023-11-02T09:22:28.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2023-11-02T09:22:29.006 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test3' already exists 2023-11-02T09:22:29.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:22:30.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test3 size 2 2023-11-02T09:22:30.871 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 3 size to 2 2023-11-02T09:22:30.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:344: TEST_recovery_pool_priority: sleep 5 2023-11-02T09:22:35.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:346: TEST_recovery_pool_priority: wait_for_clean 2023-11-02T09:22:35.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T09:22:35.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T09:22:35.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T09:22:35.886 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-11-02T09:22:35.886 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T09:22:35.886 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T09:22:35.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T09:22:35.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T09:22:35.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T09:22:35.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-11-02T09:22:35.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T09:22:35.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T09:22:35.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T09:22:35.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T09:22:35.977 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T09:22:36.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T09:22:36.387 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2023-11-02T09:22:36.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T09:22:36.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:22:36.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T09:22:36.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836487 2023-11-02T09:22:36.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836487 2023-11-02T09:22:36.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836487' 2023-11-02T09:22:36.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:22:36.503 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T09:22:36.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672965 2023-11-02T09:22:36.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672965 2023-11-02T09:22:36.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2023-11-02T09:22:36.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:22:36.616 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836487 2023-11-02T09:22:36.616 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:22:36.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T09:22:36.617 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836487 2023-11-02T09:22:36.617 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:22:36.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836487 2023-11-02T09:22:36.618 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836487 2023-11-02T09:22:36.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2023-11-02T09:22:36.619 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:22:37.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836487 -lt 21474836487 2023-11-02T09:22:37.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:22:37.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672965 2023-11-02T09:22:37.007 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:22:37.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T09:22:37.008 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672965 2023-11-02T09:22:37.008 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:22:37.008 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672965 2023-11-02T09:22:37.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672965 2023-11-02T09:22:37.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2023-11-02T09:22:37.009 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T09:22:37.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672963 -lt 42949672965 2023-11-02T09:22:37.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T09:22:38.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T09:22:38.417 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T09:22:38.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672965 -lt 42949672965 2023-11-02T09:22:38.800 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T09:22:38.800 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:22:38.801 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:22:39.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2023-11-02T09:22:39.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:22:39.306 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:22:39.306 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:22:39.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:22:39.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:22:39.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:22:39.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:22:39.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2023-11-02T09:22:39.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:22:39.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:22:39.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:22:40.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2023-11-02T09:22:40.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T09:22:40.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T09:22:40.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:348: TEST_recovery_pool_priority: ceph pg dump pgs 2023-11-02T09:22:40.619 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T09:22:40.620 INFO:tasks.workunit.client.0.smithi116.stdout:3.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-02T09:22:31.036991+0000 0'0 26:17 [1,0] 1 [1,0] 1 0'0 2023-11-02T09:22:28.659410+0000 0'0 2023-11-02T09:22:28.659410+0000 0 0 periodic scrub scheduled @ 2023-11-03T10:39:48.804290+0000 0 0 2023-11-02T09:22:40.620 INFO:tasks.workunit.client.0.smithi116.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-02T09:22:28.567542+0000 0'0 25:23 [0,1] 0 [0,1] 0 0'0 2023-11-02T09:22:26.190787+0000 0'0 2023-11-02T09:22:26.190787+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:03:27.209263+0000 0 0 2023-11-02T09:22:40.621 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-02T09:22:25.698290+0000 0'0 25:28 [1,0] 1 [1,0] 1 0'0 2023-11-02T09:22:23.428677+0000 0'0 2023-11-02T09:22:23.428677+0000 0 0 periodic scrub scheduled @ 2023-11-03T15:23:35.366499+0000 0 0 2023-11-02T09:22:40.621 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T09:22:40.621 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T09:22:40.622 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T09:22:40.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:352: TEST_recovery_pool_priority: local PG1 2023-11-02T09:22:40.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:353: TEST_recovery_pool_priority: local POOLNUM1 2023-11-02T09:22:40.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:354: TEST_recovery_pool_priority: local pool1 2023-11-02T09:22:40.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:355: TEST_recovery_pool_priority: local chk_osd1_1 2023-11-02T09:22:40.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:356: TEST_recovery_pool_priority: local chk_osd1_2 2023-11-02T09:22:40.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:358: TEST_recovery_pool_priority: local PG2 2023-11-02T09:22:40.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:359: TEST_recovery_pool_priority: local POOLNUM2 2023-11-02T09:22:40.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:360: TEST_recovery_pool_priority: local pool2 2023-11-02T09:22:40.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: local chk_osd2_1 2023-11-02T09:22:40.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:362: TEST_recovery_pool_priority: local chk_osd2_2 2023-11-02T09:22:40.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: seq 1 3 2023-11-02T09:22:40.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2023-11-02T09:22:40.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: ceph pg map 1.0 --format=json 2023-11-02T09:22:40.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: jq '.acting[]' 2023-11-02T09:22:41.061 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2023-11-02T09:22:41.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: local test_osd1=1 2023-11-02T09:22:41.062 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2023-11-02T09:22:41.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: local test_osd2=0 2023-11-02T09:22:41.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: '[' -z '' ']' 2023-11-02T09:22:41.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:371: TEST_recovery_pool_priority: PG1=1.0 2023-11-02T09:22:41.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:372: TEST_recovery_pool_priority: POOLNUM1=1 2023-11-02T09:22:41.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:373: TEST_recovery_pool_priority: pool1=test1 2023-11-02T09:22:41.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:374: TEST_recovery_pool_priority: chk_osd1_1=1 2023-11-02T09:22:41.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:375: TEST_recovery_pool_priority: chk_osd1_2=0 2023-11-02T09:22:41.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2023-11-02T09:22:41.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: ceph pg map 2.0 --format=json 2023-11-02T09:22:41.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: jq '.acting[]' 2023-11-02T09:22:41.471 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2023-11-02T09:22:41.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: local test_osd1=0 2023-11-02T09:22:41.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2023-11-02T09:22:41.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: local test_osd2=1 2023-11-02T09:22:41.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: '[' -z 1.0 ']' 2023-11-02T09:22:41.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:376: TEST_recovery_pool_priority: '[' 1 '!=' 0 ']' 2023-11-02T09:22:41.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:378: TEST_recovery_pool_priority: PG2=2.0 2023-11-02T09:22:41.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:379: TEST_recovery_pool_priority: POOLNUM2=2 2023-11-02T09:22:41.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:380: TEST_recovery_pool_priority: pool2=test2 2023-11-02T09:22:41.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:381: TEST_recovery_pool_priority: chk_osd2_1=0 2023-11-02T09:22:41.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:382: TEST_recovery_pool_priority: chk_osd2_2=1 2023-11-02T09:22:41.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:383: TEST_recovery_pool_priority: break 2023-11-02T09:22:41.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:386: TEST_recovery_pool_priority: rm -f td/osd-recovery-prio/acting 2023-11-02T09:22:41.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:388: TEST_recovery_pool_priority: '[' test2 = '' ']' 2023-11-02T09:22:41.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: seq 1 3 2023-11-02T09:22:41.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2023-11-02T09:22:41.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2023-11-02T09:22:41.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2023-11-02T09:22:41.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2023-11-02T09:22:41.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2023-11-02T09:22:41.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2023-11-02T09:22:41.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:398: TEST_recovery_pool_priority: delete_pool test3 2023-11-02T09:22:41.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2023-11-02T09:22:41.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2023-11-02T09:22:42.087 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test3' does not exist 2023-11-02T09:22:42.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:402: TEST_recovery_pool_priority: pool1_extra_prio=1 2023-11-02T09:22:42.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:403: TEST_recovery_pool_priority: pool2_extra_prio=2 2023-11-02T09:22:42.099 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: expr 190 + 1 2023-11-02T09:22:42.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: pool1_prio=191 2023-11-02T09:22:42.100 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: expr 190 + 2 2023-11-02T09:22:42.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: pool2_prio=192 2023-11-02T09:22:42.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:407: TEST_recovery_pool_priority: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2023-11-02T09:22:43.166 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 size to 1 2023-11-02T09:22:43.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:408: TEST_recovery_pool_priority: ceph osd pool set test1 recovery_priority 1 2023-11-02T09:22:44.079 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 recovery_priority to 1 2023-11-02T09:22:44.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:409: TEST_recovery_pool_priority: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2023-11-02T09:22:45.008 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 2 size to 1 2023-11-02T09:22:45.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:410: TEST_recovery_pool_priority: ceph osd pool set test2 recovery_priority 2 2023-11-02T09:22:45.838 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 2 recovery_priority to 2 2023-11-02T09:22:45.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:411: TEST_recovery_pool_priority: wait_for_clean 2023-11-02T09:22:45.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T09:22:45.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T09:22:45.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T09:22:45.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-11-02T09:22:45.852 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T09:22:45.852 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T09:22:45.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T09:22:45.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T09:22:45.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T09:22:45.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-11-02T09:22:45.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T09:22:45.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T09:22:45.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T09:22:45.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T09:22:45.931 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T09:22:46.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T09:22:46.308 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2023-11-02T09:22:46.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T09:22:46.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:22:46.309 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T09:22:46.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836491 2023-11-02T09:22:46.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836491 2023-11-02T09:22:46.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836491' 2023-11-02T09:22:46.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:22:46.425 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T09:22:46.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672969 2023-11-02T09:22:46.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672969 2023-11-02T09:22:46.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836491 1-42949672969' 2023-11-02T09:22:46.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:22:46.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836491 2023-11-02T09:22:46.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:22:46.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T09:22:46.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836491 2023-11-02T09:22:46.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:22:46.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836491 2023-11-02T09:22:46.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2023-11-02T09:22:46.555 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836491 2023-11-02T09:22:46.556 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:22:47.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836491 -lt 21474836491 2023-11-02T09:22:47.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:22:47.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672969 2023-11-02T09:22:47.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:22:47.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T09:22:47.014 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672969 2023-11-02T09:22:47.014 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:22:47.015 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672969 2023-11-02T09:22:47.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672969 2023-11-02T09:22:47.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2023-11-02T09:22:47.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T09:22:47.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672967 -lt 42949672969 2023-11-02T09:22:47.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T09:22:48.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T09:22:48.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T09:22:48.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672969 -lt 42949672969 2023-11-02T09:22:48.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T09:22:48.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:22:48.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:22:49.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2023-11-02T09:22:49.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:22:49.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:22:49.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:22:49.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:22:49.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:22:49.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:22:49.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:22:49.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:22:49.693 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:22:49.693 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:22:49.693 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:22:50.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2023-11-02T09:22:50.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T09:22:50.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T09:22:50.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:413: TEST_recovery_pool_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2023-11-02T09:22:50.262 INFO:tasks.workunit.client.0.smithi116.stderr:10+0 records in 2023-11-02T09:22:50.262 INFO:tasks.workunit.client.0.smithi116.stderr:10+0 records out 2023-11-02T09:22:50.262 INFO:tasks.workunit.client.0.smithi116.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0293968 s, 357 MB/s 2023-11-02T09:22:50.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: p=1 2023-11-02T09:22:50.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:415: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2023-11-02T09:22:50.263 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: seq 1 200 2023-11-02T09:22:50.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:50.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2023-11-02T09:22:50.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:50.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2023-11-02T09:22:51.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:51.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2023-11-02T09:22:51.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:51.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2023-11-02T09:22:51.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:51.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2023-11-02T09:22:52.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:52.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2023-11-02T09:22:52.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:52.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2023-11-02T09:22:52.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:52.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2023-11-02T09:22:53.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:53.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2023-11-02T09:22:53.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:53.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2023-11-02T09:22:53.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:53.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2023-11-02T09:22:54.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:54.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2023-11-02T09:22:54.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:54.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2023-11-02T09:22:55.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:55.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2023-11-02T09:22:55.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:55.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2023-11-02T09:22:55.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:55.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2023-11-02T09:22:56.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:56.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2023-11-02T09:22:56.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:56.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2023-11-02T09:22:56.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:56.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2023-11-02T09:22:57.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:57.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2023-11-02T09:22:57.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:57.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2023-11-02T09:22:57.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:57.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2023-11-02T09:22:58.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:58.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2023-11-02T09:22:58.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:58.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2023-11-02T09:22:58.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:58.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2023-11-02T09:22:59.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:59.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2023-11-02T09:22:59.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:59.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2023-11-02T09:22:59.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:22:59.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2023-11-02T09:23:00.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:00.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2023-11-02T09:23:00.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:00.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2023-11-02T09:23:00.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:00.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2023-11-02T09:23:01.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:01.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2023-11-02T09:23:01.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:01.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2023-11-02T09:23:01.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:01.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2023-11-02T09:23:02.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:02.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2023-11-02T09:23:02.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:02.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2023-11-02T09:23:03.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:03.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2023-11-02T09:23:03.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:03.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2023-11-02T09:23:03.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:03.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2023-11-02T09:23:04.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:04.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2023-11-02T09:23:04.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:04.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2023-11-02T09:23:04.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:04.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2023-11-02T09:23:05.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:05.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2023-11-02T09:23:05.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:05.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2023-11-02T09:23:05.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:05.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2023-11-02T09:23:06.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:06.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2023-11-02T09:23:06.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:06.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2023-11-02T09:23:07.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:07.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2023-11-02T09:23:07.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:07.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2023-11-02T09:23:07.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:07.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2023-11-02T09:23:08.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:08.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2023-11-02T09:23:08.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:08.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2023-11-02T09:23:08.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:08.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2023-11-02T09:23:09.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:09.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2023-11-02T09:23:09.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:09.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2023-11-02T09:23:09.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:09.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2023-11-02T09:23:10.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:10.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2023-11-02T09:23:10.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:10.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2023-11-02T09:23:10.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:10.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2023-11-02T09:23:11.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:11.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2023-11-02T09:23:11.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:11.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2023-11-02T09:23:11.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:11.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2023-11-02T09:23:12.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:12.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2023-11-02T09:23:12.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:12.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2023-11-02T09:23:12.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:12.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2023-11-02T09:23:13.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:13.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2023-11-02T09:23:13.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:13.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2023-11-02T09:23:13.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:13.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2023-11-02T09:23:14.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:14.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2023-11-02T09:23:14.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:14.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2023-11-02T09:23:14.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:14.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2023-11-02T09:23:15.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:15.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2023-11-02T09:23:15.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:15.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2023-11-02T09:23:15.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:15.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2023-11-02T09:23:16.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:16.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2023-11-02T09:23:16.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:16.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2023-11-02T09:23:17.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:17.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2023-11-02T09:23:17.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:17.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2023-11-02T09:23:17.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:17.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2023-11-02T09:23:18.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:18.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2023-11-02T09:23:18.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:18.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2023-11-02T09:23:18.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:18.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2023-11-02T09:23:19.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:19.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2023-11-02T09:23:19.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:19.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2023-11-02T09:23:19.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:19.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2023-11-02T09:23:20.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:20.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2023-11-02T09:23:20.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:20.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2023-11-02T09:23:20.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:20.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2023-11-02T09:23:21.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:21.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2023-11-02T09:23:21.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:21.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2023-11-02T09:23:21.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:21.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2023-11-02T09:23:22.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:22.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2023-11-02T09:23:22.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:22.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2023-11-02T09:23:23.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:23.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2023-11-02T09:23:23.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:23.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2023-11-02T09:23:23.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:23.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2023-11-02T09:23:24.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:24.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2023-11-02T09:23:24.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:24.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2023-11-02T09:23:25.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:25.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2023-11-02T09:23:25.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:25.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2023-11-02T09:23:25.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:25.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2023-11-02T09:23:25.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:25.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2023-11-02T09:23:26.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:26.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2023-11-02T09:23:26.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:26.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2023-11-02T09:23:27.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:27.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2023-11-02T09:23:27.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:27.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2023-11-02T09:23:27.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:27.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2023-11-02T09:23:28.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:28.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2023-11-02T09:23:28.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:28.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2023-11-02T09:23:28.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:28.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2023-11-02T09:23:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2023-11-02T09:23:29.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:29.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2023-11-02T09:23:29.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:29.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2023-11-02T09:23:30.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:30.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2023-11-02T09:23:30.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:30.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2023-11-02T09:23:31.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:31.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2023-11-02T09:23:31.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:31.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2023-11-02T09:23:31.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:31.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2023-11-02T09:23:32.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:32.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2023-11-02T09:23:32.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:32.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2023-11-02T09:23:32.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:32.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2023-11-02T09:23:33.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:33.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2023-11-02T09:23:33.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:33.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2023-11-02T09:23:34.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:34.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2023-11-02T09:23:34.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:34.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2023-11-02T09:23:34.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:34.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2023-11-02T09:23:35.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:35.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2023-11-02T09:23:35.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:35.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2023-11-02T09:23:35.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:35.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2023-11-02T09:23:36.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:36.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2023-11-02T09:23:36.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:36.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2023-11-02T09:23:36.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:36.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2023-11-02T09:23:37.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:37.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2023-11-02T09:23:37.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:37.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2023-11-02T09:23:37.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:37.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2023-11-02T09:23:38.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:38.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2023-11-02T09:23:38.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:38.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2023-11-02T09:23:38.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:38.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2023-11-02T09:23:39.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:39.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2023-11-02T09:23:39.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:39.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2023-11-02T09:23:40.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:40.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2023-11-02T09:23:40.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:40.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2023-11-02T09:23:40.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:40.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2023-11-02T09:23:41.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:41.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2023-11-02T09:23:41.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:41.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2023-11-02T09:23:41.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:41.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2023-11-02T09:23:42.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:42.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2023-11-02T09:23:42.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:42.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2023-11-02T09:23:42.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:42.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2023-11-02T09:23:43.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:43.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2023-11-02T09:23:43.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:43.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2023-11-02T09:23:43.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:43.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2023-11-02T09:23:44.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:44.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2023-11-02T09:23:44.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:44.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2023-11-02T09:23:45.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:45.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2023-11-02T09:23:45.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:45.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2023-11-02T09:23:45.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:45.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2023-11-02T09:23:46.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:46.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2023-11-02T09:23:46.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:46.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2023-11-02T09:23:46.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:46.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2023-11-02T09:23:47.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:47.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2023-11-02T09:23:47.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:47.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2023-11-02T09:23:47.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:47.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2023-11-02T09:23:48.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:48.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2023-11-02T09:23:48.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:48.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2023-11-02T09:23:48.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:48.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2023-11-02T09:23:49.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:49.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2023-11-02T09:23:49.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:49.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2023-11-02T09:23:49.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:49.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2023-11-02T09:23:50.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:50.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2023-11-02T09:23:50.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:50.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2023-11-02T09:23:51.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:51.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2023-11-02T09:23:51.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:51.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2023-11-02T09:23:51.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:51.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2023-11-02T09:23:52.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:52.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2023-11-02T09:23:52.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:52.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2023-11-02T09:23:52.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:52.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2023-11-02T09:23:53.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:53.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2023-11-02T09:23:53.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:53.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2023-11-02T09:23:53.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:53.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2023-11-02T09:23:54.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:54.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2023-11-02T09:23:54.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:54.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2023-11-02T09:23:55.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:55.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2023-11-02T09:23:55.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:55.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2023-11-02T09:23:55.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:55.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2023-11-02T09:23:56.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:56.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2023-11-02T09:23:56.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:56.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2023-11-02T09:23:56.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:56.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2023-11-02T09:23:57.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:57.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2023-11-02T09:23:57.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:57.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2023-11-02T09:23:57.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:57.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2023-11-02T09:23:58.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:58.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2023-11-02T09:23:58.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:58.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2023-11-02T09:23:58.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:58.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2023-11-02T09:23:59.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:59.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2023-11-02T09:23:59.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:59.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2023-11-02T09:23:59.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:23:59.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2023-11-02T09:24:00.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:00.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2023-11-02T09:24:00.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:00.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2023-11-02T09:24:00.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:00.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2023-11-02T09:24:01.199 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: expr 1 + 1 2023-11-02T09:24:01.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: p=2 2023-11-02T09:24:01.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:415: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2023-11-02T09:24:01.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: seq 1 200 2023-11-02T09:24:01.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:01.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj1-p2 td/osd-recovery-prio/data 2023-11-02T09:24:01.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:01.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj2-p2 td/osd-recovery-prio/data 2023-11-02T09:24:01.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:01.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj3-p2 td/osd-recovery-prio/data 2023-11-02T09:24:02.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:02.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj4-p2 td/osd-recovery-prio/data 2023-11-02T09:24:02.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:02.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj5-p2 td/osd-recovery-prio/data 2023-11-02T09:24:02.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:02.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj6-p2 td/osd-recovery-prio/data 2023-11-02T09:24:03.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:03.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj7-p2 td/osd-recovery-prio/data 2023-11-02T09:24:03.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:03.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj8-p2 td/osd-recovery-prio/data 2023-11-02T09:24:03.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:03.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj9-p2 td/osd-recovery-prio/data 2023-11-02T09:24:04.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:04.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj10-p2 td/osd-recovery-prio/data 2023-11-02T09:24:04.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:04.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj11-p2 td/osd-recovery-prio/data 2023-11-02T09:24:05.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:05.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj12-p2 td/osd-recovery-prio/data 2023-11-02T09:24:05.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:05.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj13-p2 td/osd-recovery-prio/data 2023-11-02T09:24:05.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:05.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj14-p2 td/osd-recovery-prio/data 2023-11-02T09:24:06.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:06.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj15-p2 td/osd-recovery-prio/data 2023-11-02T09:24:06.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:06.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj16-p2 td/osd-recovery-prio/data 2023-11-02T09:24:06.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:06.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj17-p2 td/osd-recovery-prio/data 2023-11-02T09:24:07.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:07.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj18-p2 td/osd-recovery-prio/data 2023-11-02T09:24:07.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:07.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj19-p2 td/osd-recovery-prio/data 2023-11-02T09:24:07.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:07.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj20-p2 td/osd-recovery-prio/data 2023-11-02T09:24:08.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:08.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj21-p2 td/osd-recovery-prio/data 2023-11-02T09:24:08.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:08.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj22-p2 td/osd-recovery-prio/data 2023-11-02T09:24:08.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:08.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj23-p2 td/osd-recovery-prio/data 2023-11-02T09:24:09.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:09.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj24-p2 td/osd-recovery-prio/data 2023-11-02T09:24:09.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:09.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj25-p2 td/osd-recovery-prio/data 2023-11-02T09:24:09.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:09.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj26-p2 td/osd-recovery-prio/data 2023-11-02T09:24:10.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:10.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj27-p2 td/osd-recovery-prio/data 2023-11-02T09:24:10.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:10.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj28-p2 td/osd-recovery-prio/data 2023-11-02T09:24:11.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:11.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj29-p2 td/osd-recovery-prio/data 2023-11-02T09:24:11.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:11.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj30-p2 td/osd-recovery-prio/data 2023-11-02T09:24:11.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:11.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj31-p2 td/osd-recovery-prio/data 2023-11-02T09:24:12.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:12.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj32-p2 td/osd-recovery-prio/data 2023-11-02T09:24:12.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:12.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj33-p2 td/osd-recovery-prio/data 2023-11-02T09:24:12.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:12.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj34-p2 td/osd-recovery-prio/data 2023-11-02T09:24:13.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:13.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj35-p2 td/osd-recovery-prio/data 2023-11-02T09:24:13.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:13.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj36-p2 td/osd-recovery-prio/data 2023-11-02T09:24:13.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:13.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj37-p2 td/osd-recovery-prio/data 2023-11-02T09:24:14.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:14.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj38-p2 td/osd-recovery-prio/data 2023-11-02T09:24:14.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:14.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj39-p2 td/osd-recovery-prio/data 2023-11-02T09:24:15.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:15.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj40-p2 td/osd-recovery-prio/data 2023-11-02T09:24:15.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:15.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj41-p2 td/osd-recovery-prio/data 2023-11-02T09:24:15.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:15.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj42-p2 td/osd-recovery-prio/data 2023-11-02T09:24:16.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:16.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj43-p2 td/osd-recovery-prio/data 2023-11-02T09:24:16.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:16.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj44-p2 td/osd-recovery-prio/data 2023-11-02T09:24:16.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:16.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj45-p2 td/osd-recovery-prio/data 2023-11-02T09:24:17.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:17.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj46-p2 td/osd-recovery-prio/data 2023-11-02T09:24:17.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:17.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj47-p2 td/osd-recovery-prio/data 2023-11-02T09:24:17.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:17.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj48-p2 td/osd-recovery-prio/data 2023-11-02T09:24:18.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:18.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj49-p2 td/osd-recovery-prio/data 2023-11-02T09:24:18.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:18.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj50-p2 td/osd-recovery-prio/data 2023-11-02T09:24:18.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:18.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj51-p2 td/osd-recovery-prio/data 2023-11-02T09:24:19.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:19.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj52-p2 td/osd-recovery-prio/data 2023-11-02T09:24:19.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:19.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj53-p2 td/osd-recovery-prio/data 2023-11-02T09:24:19.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:19.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj54-p2 td/osd-recovery-prio/data 2023-11-02T09:24:20.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:20.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj55-p2 td/osd-recovery-prio/data 2023-11-02T09:24:20.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:20.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj56-p2 td/osd-recovery-prio/data 2023-11-02T09:24:21.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:21.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj57-p2 td/osd-recovery-prio/data 2023-11-02T09:24:21.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:21.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj58-p2 td/osd-recovery-prio/data 2023-11-02T09:24:21.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:21.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj59-p2 td/osd-recovery-prio/data 2023-11-02T09:24:22.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:22.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj60-p2 td/osd-recovery-prio/data 2023-11-02T09:24:22.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:22.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj61-p2 td/osd-recovery-prio/data 2023-11-02T09:24:22.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:22.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj62-p2 td/osd-recovery-prio/data 2023-11-02T09:24:23.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:23.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj63-p2 td/osd-recovery-prio/data 2023-11-02T09:24:23.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:23.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj64-p2 td/osd-recovery-prio/data 2023-11-02T09:24:23.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:23.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj65-p2 td/osd-recovery-prio/data 2023-11-02T09:24:24.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:24.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj66-p2 td/osd-recovery-prio/data 2023-11-02T09:24:24.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:24.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj67-p2 td/osd-recovery-prio/data 2023-11-02T09:24:24.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:24.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj68-p2 td/osd-recovery-prio/data 2023-11-02T09:24:25.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:25.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj69-p2 td/osd-recovery-prio/data 2023-11-02T09:24:25.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:25.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj70-p2 td/osd-recovery-prio/data 2023-11-02T09:24:25.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:25.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj71-p2 td/osd-recovery-prio/data 2023-11-02T09:24:26.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:26.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj72-p2 td/osd-recovery-prio/data 2023-11-02T09:24:26.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:26.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj73-p2 td/osd-recovery-prio/data 2023-11-02T09:24:26.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:26.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj74-p2 td/osd-recovery-prio/data 2023-11-02T09:24:27.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:27.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj75-p2 td/osd-recovery-prio/data 2023-11-02T09:24:27.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:27.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj76-p2 td/osd-recovery-prio/data 2023-11-02T09:24:27.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:27.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj77-p2 td/osd-recovery-prio/data 2023-11-02T09:24:28.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:28.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj78-p2 td/osd-recovery-prio/data 2023-11-02T09:24:28.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:28.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj79-p2 td/osd-recovery-prio/data 2023-11-02T09:24:29.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:29.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj80-p2 td/osd-recovery-prio/data 2023-11-02T09:24:29.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:29.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj81-p2 td/osd-recovery-prio/data 2023-11-02T09:24:29.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:29.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj82-p2 td/osd-recovery-prio/data 2023-11-02T09:24:30.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:30.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj83-p2 td/osd-recovery-prio/data 2023-11-02T09:24:30.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:30.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj84-p2 td/osd-recovery-prio/data 2023-11-02T09:24:30.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:30.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj85-p2 td/osd-recovery-prio/data 2023-11-02T09:24:31.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:31.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj86-p2 td/osd-recovery-prio/data 2023-11-02T09:24:31.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:31.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj87-p2 td/osd-recovery-prio/data 2023-11-02T09:24:31.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:31.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj88-p2 td/osd-recovery-prio/data 2023-11-02T09:24:32.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:32.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj89-p2 td/osd-recovery-prio/data 2023-11-02T09:24:32.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:32.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj90-p2 td/osd-recovery-prio/data 2023-11-02T09:24:32.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:32.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj91-p2 td/osd-recovery-prio/data 2023-11-02T09:24:33.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:33.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj92-p2 td/osd-recovery-prio/data 2023-11-02T09:24:33.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:33.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj93-p2 td/osd-recovery-prio/data 2023-11-02T09:24:34.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:34.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj94-p2 td/osd-recovery-prio/data 2023-11-02T09:24:34.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:34.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj95-p2 td/osd-recovery-prio/data 2023-11-02T09:24:34.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:34.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj96-p2 td/osd-recovery-prio/data 2023-11-02T09:24:35.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:35.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj97-p2 td/osd-recovery-prio/data 2023-11-02T09:24:35.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:35.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj98-p2 td/osd-recovery-prio/data 2023-11-02T09:24:35.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:35.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj99-p2 td/osd-recovery-prio/data 2023-11-02T09:24:36.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:36.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj100-p2 td/osd-recovery-prio/data 2023-11-02T09:24:36.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:36.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj101-p2 td/osd-recovery-prio/data 2023-11-02T09:24:36.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:36.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj102-p2 td/osd-recovery-prio/data 2023-11-02T09:24:37.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:37.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj103-p2 td/osd-recovery-prio/data 2023-11-02T09:24:37.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:37.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj104-p2 td/osd-recovery-prio/data 2023-11-02T09:24:38.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:38.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj105-p2 td/osd-recovery-prio/data 2023-11-02T09:24:38.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:38.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj106-p2 td/osd-recovery-prio/data 2023-11-02T09:24:38.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:38.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj107-p2 td/osd-recovery-prio/data 2023-11-02T09:24:39.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:39.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj108-p2 td/osd-recovery-prio/data 2023-11-02T09:24:39.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:39.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj109-p2 td/osd-recovery-prio/data 2023-11-02T09:24:39.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:39.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj110-p2 td/osd-recovery-prio/data 2023-11-02T09:24:40.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:40.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj111-p2 td/osd-recovery-prio/data 2023-11-02T09:24:40.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:40.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj112-p2 td/osd-recovery-prio/data 2023-11-02T09:24:40.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:40.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj113-p2 td/osd-recovery-prio/data 2023-11-02T09:24:41.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:41.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj114-p2 td/osd-recovery-prio/data 2023-11-02T09:24:41.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:41.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj115-p2 td/osd-recovery-prio/data 2023-11-02T09:24:41.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:41.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj116-p2 td/osd-recovery-prio/data 2023-11-02T09:24:42.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:42.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj117-p2 td/osd-recovery-prio/data 2023-11-02T09:24:42.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:42.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj118-p2 td/osd-recovery-prio/data 2023-11-02T09:24:43.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:43.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj119-p2 td/osd-recovery-prio/data 2023-11-02T09:24:43.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:43.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj120-p2 td/osd-recovery-prio/data 2023-11-02T09:24:43.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:43.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj121-p2 td/osd-recovery-prio/data 2023-11-02T09:24:44.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:44.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj122-p2 td/osd-recovery-prio/data 2023-11-02T09:24:44.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:44.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj123-p2 td/osd-recovery-prio/data 2023-11-02T09:24:44.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:44.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj124-p2 td/osd-recovery-prio/data 2023-11-02T09:24:45.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:45.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj125-p2 td/osd-recovery-prio/data 2023-11-02T09:24:45.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:45.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj126-p2 td/osd-recovery-prio/data 2023-11-02T09:24:45.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:45.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj127-p2 td/osd-recovery-prio/data 2023-11-02T09:24:46.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:46.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj128-p2 td/osd-recovery-prio/data 2023-11-02T09:24:46.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:46.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj129-p2 td/osd-recovery-prio/data 2023-11-02T09:24:46.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:46.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj130-p2 td/osd-recovery-prio/data 2023-11-02T09:24:47.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:47.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj131-p2 td/osd-recovery-prio/data 2023-11-02T09:24:47.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:47.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj132-p2 td/osd-recovery-prio/data 2023-11-02T09:24:48.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:48.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj133-p2 td/osd-recovery-prio/data 2023-11-02T09:24:48.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:48.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj134-p2 td/osd-recovery-prio/data 2023-11-02T09:24:48.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:48.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj135-p2 td/osd-recovery-prio/data 2023-11-02T09:24:49.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:49.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj136-p2 td/osd-recovery-prio/data 2023-11-02T09:24:49.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:49.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj137-p2 td/osd-recovery-prio/data 2023-11-02T09:24:49.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:49.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj138-p2 td/osd-recovery-prio/data 2023-11-02T09:24:50.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:50.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj139-p2 td/osd-recovery-prio/data 2023-11-02T09:24:50.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:50.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj140-p2 td/osd-recovery-prio/data 2023-11-02T09:24:50.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:50.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj141-p2 td/osd-recovery-prio/data 2023-11-02T09:24:51.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:51.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj142-p2 td/osd-recovery-prio/data 2023-11-02T09:24:51.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:51.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj143-p2 td/osd-recovery-prio/data 2023-11-02T09:24:51.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:51.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj144-p2 td/osd-recovery-prio/data 2023-11-02T09:24:52.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:52.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj145-p2 td/osd-recovery-prio/data 2023-11-02T09:24:52.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:52.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj146-p2 td/osd-recovery-prio/data 2023-11-02T09:24:53.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:53.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj147-p2 td/osd-recovery-prio/data 2023-11-02T09:24:53.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:53.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj148-p2 td/osd-recovery-prio/data 2023-11-02T09:24:53.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:53.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj149-p2 td/osd-recovery-prio/data 2023-11-02T09:24:54.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:54.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj150-p2 td/osd-recovery-prio/data 2023-11-02T09:24:54.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:54.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj151-p2 td/osd-recovery-prio/data 2023-11-02T09:24:54.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:54.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj152-p2 td/osd-recovery-prio/data 2023-11-02T09:24:55.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:55.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj153-p2 td/osd-recovery-prio/data 2023-11-02T09:24:55.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:55.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj154-p2 td/osd-recovery-prio/data 2023-11-02T09:24:55.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:55.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj155-p2 td/osd-recovery-prio/data 2023-11-02T09:24:56.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:56.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj156-p2 td/osd-recovery-prio/data 2023-11-02T09:24:56.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:56.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj157-p2 td/osd-recovery-prio/data 2023-11-02T09:24:57.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:57.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj158-p2 td/osd-recovery-prio/data 2023-11-02T09:24:57.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:57.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj159-p2 td/osd-recovery-prio/data 2023-11-02T09:24:57.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:57.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj160-p2 td/osd-recovery-prio/data 2023-11-02T09:24:58.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:58.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj161-p2 td/osd-recovery-prio/data 2023-11-02T09:24:58.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:58.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj162-p2 td/osd-recovery-prio/data 2023-11-02T09:24:58.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:58.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj163-p2 td/osd-recovery-prio/data 2023-11-02T09:24:59.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:59.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj164-p2 td/osd-recovery-prio/data 2023-11-02T09:24:59.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:59.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj165-p2 td/osd-recovery-prio/data 2023-11-02T09:24:59.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:24:59.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj166-p2 td/osd-recovery-prio/data 2023-11-02T09:25:00.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:00.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj167-p2 td/osd-recovery-prio/data 2023-11-02T09:25:00.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:00.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj168-p2 td/osd-recovery-prio/data 2023-11-02T09:25:00.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:00.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj169-p2 td/osd-recovery-prio/data 2023-11-02T09:25:01.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:01.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj170-p2 td/osd-recovery-prio/data 2023-11-02T09:25:01.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:01.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj171-p2 td/osd-recovery-prio/data 2023-11-02T09:25:01.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:01.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj172-p2 td/osd-recovery-prio/data 2023-11-02T09:25:02.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:02.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj173-p2 td/osd-recovery-prio/data 2023-11-02T09:25:02.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:02.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj174-p2 td/osd-recovery-prio/data 2023-11-02T09:25:03.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:03.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj175-p2 td/osd-recovery-prio/data 2023-11-02T09:25:03.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:03.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj176-p2 td/osd-recovery-prio/data 2023-11-02T09:25:03.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:03.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj177-p2 td/osd-recovery-prio/data 2023-11-02T09:25:04.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:04.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj178-p2 td/osd-recovery-prio/data 2023-11-02T09:25:04.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:04.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj179-p2 td/osd-recovery-prio/data 2023-11-02T09:25:05.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:05.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj180-p2 td/osd-recovery-prio/data 2023-11-02T09:25:05.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:05.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj181-p2 td/osd-recovery-prio/data 2023-11-02T09:25:05.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:05.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj182-p2 td/osd-recovery-prio/data 2023-11-02T09:25:06.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:06.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj183-p2 td/osd-recovery-prio/data 2023-11-02T09:25:06.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:06.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj184-p2 td/osd-recovery-prio/data 2023-11-02T09:25:06.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:06.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj185-p2 td/osd-recovery-prio/data 2023-11-02T09:25:07.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:07.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj186-p2 td/osd-recovery-prio/data 2023-11-02T09:25:07.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:07.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj187-p2 td/osd-recovery-prio/data 2023-11-02T09:25:07.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:07.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj188-p2 td/osd-recovery-prio/data 2023-11-02T09:25:08.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:08.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj189-p2 td/osd-recovery-prio/data 2023-11-02T09:25:08.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:08.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj190-p2 td/osd-recovery-prio/data 2023-11-02T09:25:08.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:08.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj191-p2 td/osd-recovery-prio/data 2023-11-02T09:25:09.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:09.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj192-p2 td/osd-recovery-prio/data 2023-11-02T09:25:09.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:09.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj193-p2 td/osd-recovery-prio/data 2023-11-02T09:25:09.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:09.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj194-p2 td/osd-recovery-prio/data 2023-11-02T09:25:10.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:10.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj195-p2 td/osd-recovery-prio/data 2023-11-02T09:25:10.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:10.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj196-p2 td/osd-recovery-prio/data 2023-11-02T09:25:11.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:11.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj197-p2 td/osd-recovery-prio/data 2023-11-02T09:25:11.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:11.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj198-p2 td/osd-recovery-prio/data 2023-11-02T09:25:11.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:11.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj199-p2 td/osd-recovery-prio/data 2023-11-02T09:25:12.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2023-11-02T09:25:12.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj200-p2 td/osd-recovery-prio/data 2023-11-02T09:25:12.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: expr 2 + 1 2023-11-02T09:25:12.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: p=3 2023-11-02T09:25:12.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: get_not_primary test1 obj1-p1 2023-11-02T09:25:12.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test1 2023-11-02T09:25:12.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1-p1 2023-11-02T09:25:12.422 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test1 obj1-p1 2023-11-02T09:25:12.422 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test1 2023-11-02T09:25:12.422 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1-p1 2023-11-02T09:25:12.422 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test1 obj1-p1 2023-11-02T09:25:12.423 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2023-11-02T09:25:12.805 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2023-11-02T09:25:12.805 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test1 obj1-p1 2023-11-02T09:25:12.805 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2023-11-02T09:25:13.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: local otherosd=null 2023-11-02T09:25:13.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:426: TEST_recovery_pool_priority: ceph pg dump pgs 2023-11-02T09:25:13.545 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T09:25:13.545 INFO:tasks.workunit.client.0.smithi116.stdout:2.0 188 0 0 0 0 1969225728 0 0 563 0 563 active+clean 2023-11-02T09:22:45.442315+0000 37'563 37:604 [0] 0 [0] 0 0'0 2023-11-02T09:22:26.190787+0000 0'0 2023-11-02T09:22:26.190787+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:42:44.365076+0000 0 0 2023-11-02T09:25:13.545 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2023-11-02T09:22:43.046362+0000 37'600 37:646 [1] 1 [1] 1 0'0 2023-11-02T09:22:23.428677+0000 0'0 2023-11-02T09:22:23.428677+0000 0 0 periodic scrub scheduled @ 2023-11-03T12:17:19.177305+0000 0 0 2023-11-02T09:25:13.546 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T09:25:13.546 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T09:25:13.546 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T09:25:13.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:427: TEST_recovery_pool_priority: ERRORS=0 2023-11-02T09:25:13.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:429: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2023-11-02T09:25:14.321 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 size to 2 2023-11-02T09:25:14.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:430: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2023-11-02T09:25:15.662 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 2 size to 2 2023-11-02T09:25:15.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:433: TEST_recovery_pool_priority: ceph pg dump pgs 2023-11-02T09:25:16.054 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T09:25:16.054 INFO:tasks.workunit.client.0.smithi116.stdout:2.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2023-11-02T09:22:45.442315+0000 37'600 37:641 [0] 0 [0] 0 0'0 2023-11-02T09:22:26.190787+0000 0'0 2023-11-02T09:22:26.190787+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:42:44.365076+0000 0 0 2023-11-02T09:25:16.055 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2023-11-02T09:22:43.046362+0000 37'600 37:646 [1] 1 [1] 1 0'0 2023-11-02T09:22:23.428677+0000 0'0 2023-11-02T09:22:23.428677+0000 0 0 periodic scrub scheduled @ 2023-11-03T12:17:19.177305+0000 0 0 2023-11-02T09:25:16.055 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T09:25:16.055 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T09:25:16.055 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T09:25:16.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:436: TEST_recovery_pool_priority: set -o pipefail 2023-11-02T09:25:16.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:437: TEST_recovery_pool_priority: count=0 2023-11-02T09:25:16.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2023-11-02T09:25:16.067 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2023-11-02T09:25:16.068 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2023-11-02T09:25:16.068 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2023-11-02T09:25:16.422 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T09:25:16.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 0 == 2 2023-11-02T09:25:16.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:445: TEST_recovery_pool_priority: sleep 2 2023-11-02T09:25:18.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:446: TEST_recovery_pool_priority: test 0 -eq 10 2023-11-02T09:25:18.434 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: expr 0 + 1 2023-11-02T09:25:18.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: count=1 2023-11-02T09:25:18.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2023-11-02T09:25:18.437 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2023-11-02T09:25:18.437 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2023-11-02T09:25:18.437 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2023-11-02T09:25:18.801 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T09:25:18.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 1 == 2 2023-11-02T09:25:18.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:445: TEST_recovery_pool_priority: sleep 2 2023-11-02T09:25:20.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:446: TEST_recovery_pool_priority: test 1 -eq 10 2023-11-02T09:25:20.814 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: expr 1 + 1 2023-11-02T09:25:20.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: count=2 2023-11-02T09:25:20.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2023-11-02T09:25:20.816 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2023-11-02T09:25:20.817 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2023-11-02T09:25:20.817 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2023-11-02T09:25:21.215 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T09:25:21.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 2 == 2 2023-11-02T09:25:21.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: break 2023-11-02T09:25:21.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: set +o pipefail 2023-11-02T09:25:21.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:454: TEST_recovery_pool_priority: ceph pg dump pgs 2023-11-02T09:25:21.631 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T09:25:21.632 INFO:tasks.workunit.client.0.smithi116.stdout:2.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+undersized+degraded+remapped 2023-11-02T09:25:17.843585+0000 37'600 45:656 [0,1] 0 [0] 0 0'0 2023-11-02T09:22:26.190787+0000 0'0 2023-11-02T09:22:26.190787+0000 0 0 periodic scrub scheduled @ 2023-11-03T20:52:30.174899+0000 0 0 2023-11-02T09:25:21.632 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 200 0 194 0 0 2097152000 0 0 600 0 600 active+recovering+undersized+degraded+remapped 2023-11-02T09:25:15.576075+0000 37'600 45:673 [1,0] 1 [1] 1 0'0 2023-11-02T09:22:23.428677+0000 0'0 2023-11-02T09:22:23.428677+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:04:34.929068+0000 0 0 2023-11-02T09:25:21.633 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T09:25:21.633 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T09:25:21.633 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T09:25:21.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: get_asok_path osd.1 2023-11-02T09:25:21.644 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-11-02T09:25:21.644 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-11-02T09:25:21.645 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:25:21.645 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:25:21.645 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:25:21.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.261297/ceph-osd.1.asok 2023-11-02T09:25:21.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: CEPH_ARGS= 2023-11-02T09:25:21.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:25:21.728 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 2023-11-02T09:25:21.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:457: TEST_recovery_pool_priority: echo osd.1 2023-11-02T09:25:21.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:458: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2023-11-02T09:25:21.730 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:25:21.730 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:25:21.730 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:25:21.731 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:25:21.731 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:25:21.731 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:25:21.731 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:25:21.732 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:25:21.732 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 191, 2023-11-02T09:25:21.732 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:25:21.732 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:25:21.732 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:25:21.733 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:25:21.733 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:25:21.733 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:25:21.733 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:25:21.733 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:25:21.734 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:25:21.734 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:25:21.734 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "2.0", 2023-11-02T09:25:21.735 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 192, 2023-11-02T09:25:21.735 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:25:21.735 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:25:21.736 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:25:21.736 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:25:21.737 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:25:21.738 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: get_asok_path osd.0 2023-11-02T09:25:21.739 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-11-02T09:25:21.739 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-11-02T09:25:21.739 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:25:21.739 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:25:21.739 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:25:21.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.261297/ceph-osd.0.asok 2023-11-02T09:25:21.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: CEPH_ARGS= 2023-11-02T09:25:21.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.0.asok dump_recovery_reservations 2023-11-02T09:25:21.830 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 2023-11-02T09:25:21.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:460: TEST_recovery_pool_priority: echo osd.0 2023-11-02T09:25:21.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2023-11-02T09:25:21.831 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:25:21.832 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:25:21.832 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:25:21.833 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:25:21.833 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:25:21.833 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:25:21.834 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:25:21.834 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "2.0", 2023-11-02T09:25:21.834 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 192, 2023-11-02T09:25:21.835 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:25:21.835 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:25:21.835 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:25:21.835 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:25:21.836 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:25:21.836 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:25:21.836 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:25:21.836 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:25:21.837 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:25:21.837 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:25:21.837 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:25:21.838 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 191, 2023-11-02T09:25:21.839 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:25:21.839 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:25:21.839 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:25:21.840 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:25:21.840 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:25:21.840 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2023-11-02T09:25:21.841 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2023-11-02T09:25:21.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2023-11-02T09:25:21.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: ITEM=1.0 2023-11-02T09:25:21.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:465: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2023-11-02T09:25:21.859 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2023-11-02T09:25:21.859 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2023-11-02T09:25:21.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: PRIO=191 2023-11-02T09:25:21.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:471: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2023-11-02T09:25:21.886 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2023-11-02T09:25:21.886 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2023-11-02T09:25:21.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2023-11-02T09:25:21.913 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: ITEM=1.0 2023-11-02T09:25:21.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:480: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2023-11-02T09:25:21.914 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2023-11-02T09:25:21.914 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2023-11-02T09:25:21.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: PRIO=191 2023-11-02T09:25:21.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:486: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2023-11-02T09:25:21.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2023-11-02T09:25:21.941 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2023-11-02T09:25:21.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2023-11-02T09:25:21.967 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: ITEM=2.0 2023-11-02T09:25:21.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:495: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2023-11-02T09:25:21.968 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2023-11-02T09:25:21.968 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2023-11-02T09:25:21.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: PRIO=192 2023-11-02T09:25:21.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:501: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2023-11-02T09:25:21.995 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2023-11-02T09:25:21.996 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2023-11-02T09:25:22.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2023-11-02T09:25:22.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: ITEM=2.0 2023-11-02T09:25:22.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:510: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2023-11-02T09:25:22.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2023-11-02T09:25:22.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2023-11-02T09:25:22.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: PRIO=192 2023-11-02T09:25:22.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:516: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2023-11-02T09:25:22.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:523: TEST_recovery_pool_priority: wait_for_clean 2023-11-02T09:25:22.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T09:25:22.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T09:25:22.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T09:25:22.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-11-02T09:25:22.053 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T09:25:22.053 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T09:25:22.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T09:25:22.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T09:25:22.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T09:25:22.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-11-02T09:25:22.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T09:25:22.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T09:25:22.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T09:25:22.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T09:25:22.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T09:25:22.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T09:25:22.626 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2023-11-02T09:25:22.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T09:25:22.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:25:22.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T09:25:23.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836524 2023-11-02T09:25:23.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836524 2023-11-02T09:25:23.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836524' 2023-11-02T09:25:23.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:25:23.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T09:25:23.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673002 2023-11-02T09:25:23.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673002 2023-11-02T09:25:23.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836524 1-42949673002' 2023-11-02T09:25:23.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:25:23.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836524 2023-11-02T09:25:23.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:25:23.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T09:25:23.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836524 2023-11-02T09:25:23.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:25:23.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836524 2023-11-02T09:25:23.221 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836524 2023-11-02T09:25:23.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2023-11-02T09:25:23.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:25:23.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836522 -lt 21474836524 2023-11-02T09:25:23.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T09:25:24.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T09:25:24.931 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:25:25.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836525 -lt 21474836524 2023-11-02T09:25:25.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:25:25.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673002 2023-11-02T09:25:25.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:25:25.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T09:25:25.583 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673002 2023-11-02T09:25:25.583 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:25:25.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673002 2023-11-02T09:25:25.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673002' 2023-11-02T09:25:25.584 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949673002 2023-11-02T09:25:25.584 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T09:25:26.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673003 -lt 42949673002 2023-11-02T09:25:26.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T09:25:26.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:25:26.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:25:27.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2023-11-02T09:25:27.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:25:27.169 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:25:27.169 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:25:27.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:25:27.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:25:27.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:25:27.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:25:27.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:25:27.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:25:27.809 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:25:27.809 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:25:28.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:25:28.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2023-11-02T09:25:28.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2023-11-02T09:25:28.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2023-11-02T09:25:28.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:25:28.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:25:28.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:25:28.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:25:28.414 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:25:28.414 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:25:28.414 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:25:28.414 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:25:28.415 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:25:28.415 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:25:28.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:25:28.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:25:28.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:25:28.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:25:29.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:25:29.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:25:29.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:25:29.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:25:29.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:25:29.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:25:29.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:25:29.611 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:25:29.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:25:30.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27430998 2023-11-02T09:25:30.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27430998 '!=' null 2023-11-02T09:25:30.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:25:30.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:25:30.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:25:30.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:25:30.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:25:30.241 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:25:30.241 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:25:30.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:25:30.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:25:30.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:25:30.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:25:30.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:25:30.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:25:30.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:25:30.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:25:31.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:25:31.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:25:31.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:25:31.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:25:31.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:25:31.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:25:31.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:25:31.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:25:31.308 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:25:31.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34945885 2023-11-02T09:25:31.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34945885 '!=' null 2023-11-02T09:25:31.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:25:31.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:25:31.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:25:32.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:25:32.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:25:32.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:25:32.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:25:32.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:25:32.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:25:32.058 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:25:32.058 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:25:32.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:25:32.423 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:25:32.423 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:25:32.423 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:25:32.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:25:32.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:25:32.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:25:32.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:25:32.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:25:32.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:25:32.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:25:32.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:25:32.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:25:33.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29704246 2023-11-02T09:25:33.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29704246 '!=' null 2023-11-02T09:25:33.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:25:33.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:25:33.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:25:33.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:25:33.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:25:33.893 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:25:33.893 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:25:33.894 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:25:33.894 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:25:33.894 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:25:33.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:25:34.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:25:34.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:25:34.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:25:34.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:25:34.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:25:34.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:25:34.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:25:34.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:25:34.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:25:34.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:25:34.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:25:34.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:25:34.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:25:35.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=41935725 2023-11-02T09:25:35.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 41935725 '!=' null 2023-11-02T09:25:35.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:25:35.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:25:35.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:25:35.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:25:35.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:25:35.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:25:35.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:25:35.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:25:35.616 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:25:35.616 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:25:35.616 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:25:36.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:25:36.066 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:25:36.066 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:25:36.066 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:25:36.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:25:36.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:25:36.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:25:36.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:25:36.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:25:36.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:25:36.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:25:36.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:25:36.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:25:37.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209683 2023-11-02T09:25:37.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209683 '!=' null 2023-11-02T09:25:37.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:25:37.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:25:37.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:25:37.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:25:37.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:25:37.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:25:37.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:25:37.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:25:37.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:25:37.409 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:25:37.409 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:25:37.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:25:37.856 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:25:37.856 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:25:37.856 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:25:38.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:25:38.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:25:38.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:25:38.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:25:38.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:25:38.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:25:38.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:25:38.369 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:25:38.369 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:25:39.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=32325653 2023-11-02T09:25:39.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 32325653 '!=' null 2023-11-02T09:25:39.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:25:39.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:25:39.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:25:39.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:25:39.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:25:39.355 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:25:39.355 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:25:39.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:25:39.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:25:39.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:25:39.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:25:39.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:25:39.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:25:39.741 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:25:39.741 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:25:40.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:25:40.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:25:40.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:25:40.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:25:40.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:25:40.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:25:40.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:25:40.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:25:40.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:25:40.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34072927 2023-11-02T09:25:40.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34072927 '!=' null 2023-11-02T09:25:40.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:25:40.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:25:40.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:25:41.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:25:41.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:25:41.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:25:41.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:25:41.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:25:41.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:25:41.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:25:41.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:25:41.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:25:41.543 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:25:41.543 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:25:41.544 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:25:42.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:25:42.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:25:42.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:25:42.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:25:42.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:25:42.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:25:42.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:25:42.190 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:25:42.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:25:42.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34072927 2023-11-02T09:25:42.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34072927 '!=' null 2023-11-02T09:25:42.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:25:42.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:25:42.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:25:42.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:25:42.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:25:42.832 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:25:42.832 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:25:42.833 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:25:42.833 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:25:42.833 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:25:42.833 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:25:43.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:25:43.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:25:43.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:25:43.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:25:43.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:25:43.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:25:44.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:25:44.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:25:44.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:25:44.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:25:44.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:25:44.001 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:25:44.001 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:25:44.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336250 2023-11-02T09:25:44.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336250 '!=' null 2023-11-02T09:25:44.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:25:44.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:25:44.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:25:44.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:25:44.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:25:44.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:25:44.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:25:44.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:25:44.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:25:44.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:25:44.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:25:45.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:25:45.236 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:25:45.237 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:25:45.237 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:25:45.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:25:45.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:25:45.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:25:45.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:25:45.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:25:45.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:25:45.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:25:45.921 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:25:45.921 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:25:46.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=36694031 2023-11-02T09:25:46.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 36694031 '!=' null 2023-11-02T09:25:46.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:25:46.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:25:46.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:25:46.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:25:46.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:25:46.566 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:25:46.566 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:25:46.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:25:46.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:25:46.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:25:46.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:25:46.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:25:46.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:25:46.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:25:46.937 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:25:47.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:25:47.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:25:47.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:25:47.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:25:47.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:25:47.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:25:47.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:25:47.575 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:25:47.575 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:25:48.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462502 2023-11-02T09:25:48.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462502 '!=' null 2023-11-02T09:25:48.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:25:48.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:25:48.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:25:48.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:25:48.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:25:48.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:25:48.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:25:48.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:25:48.429 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:25:48.429 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:25:48.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:25:48.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:25:48.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:25:48.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:25:48.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:25:49.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:25:49.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:25:49.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:25:49.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:25:49.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:25:49.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:25:49.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:25:49.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:25:49.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:25:50.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30578406 2023-11-02T09:25:50.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30578406 '!=' null 2023-11-02T09:25:50.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:25:50.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:25:50.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:25:50.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:25:50.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:25:50.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:25:50.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:25:50.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:25:50.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:25:50.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:25:50.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:25:50.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:25:50.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:25:50.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:25:50.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:25:51.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:25:51.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:25:51.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:25:51.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:25:51.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:25:51.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:25:51.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:25:51.488 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:25:51.489 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:25:52.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30578321 2023-11-02T09:25:52.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30578321 '!=' null 2023-11-02T09:25:52.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:25:52.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:25:52.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:25:52.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:25:52.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:25:52.234 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:25:52.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:25:52.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:25:52.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:25:52.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:25:52.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:25:52.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:25:52.660 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:25:52.660 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:25:52.660 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:25:53.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:25:53.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:25:53.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:25:53.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:25:53.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:25:53.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:25:53.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:25:53.332 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:25:53.332 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:25:54.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588898 2023-11-02T09:25:54.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588898 '!=' null 2023-11-02T09:25:54.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:25:54.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:25:54.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:25:54.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:25:54.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:25:54.217 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:25:54.217 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:25:54.217 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:25:54.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:25:54.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:25:54.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:25:54.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:25:54.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:25:54.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:25:54.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:25:55.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:25:55.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:25:55.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:25:55.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:25:55.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:25:55.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:25:55.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:25:55.157 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:25:55.158 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:25:55.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=35820251 2023-11-02T09:25:55.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 35820251 '!=' null 2023-11-02T09:25:55.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:25:55.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:25:55.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:25:55.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:25:55.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:25:55.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:25:55.947 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:25:55.947 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:25:55.947 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:25:55.947 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:25:55.947 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:25:56.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:25:56.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:25:56.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:25:56.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:25:56.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:25:56.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:25:56.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:25:56.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:25:56.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:25:56.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:25:56.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:25:56.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:25:56.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:25:57.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462400 2023-11-02T09:25:57.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462400 '!=' null 2023-11-02T09:25:57.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:25:57.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:25:57.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:25:57.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:25:57.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:25:57.764 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:25:57.765 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:25:57.765 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:25:57.766 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:25:57.766 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:25:57.766 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:25:58.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:25:58.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:25:58.133 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:25:58.134 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:25:58.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:25:58.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:25:58.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:25:58.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:25:58.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:25:58.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:25:58.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:25:58.639 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:25:58.640 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:25:59.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29704561 2023-11-02T09:25:59.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29704561 '!=' null 2023-11-02T09:25:59.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:25:59.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:25:59.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:25:59.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:25:59.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:25:59.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:25:59.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:25:59.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:25:59.248 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:25:59.248 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:25:59.248 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:25:59.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:25:59.617 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:25:59.617 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:25:59.617 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:00.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:00.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:00.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:00.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:00.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:00.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:00.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:00.382 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:00.382 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:01.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29704701 2023-11-02T09:26:01.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29704701 '!=' null 2023-11-02T09:26:01.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:01.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:01.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:01.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:01.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:01.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:01.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:01.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:01.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:01.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:01.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:01.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:01.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:01.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:01.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:02.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:02.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:02.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:02.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:02.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:02.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:02.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:02.062 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:02.063 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:02.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29704701 2023-11-02T09:26:02.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29704701 '!=' null 2023-11-02T09:26:02.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:02.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:02.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:02.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:02.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:02.913 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:02.913 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:02.914 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:02.914 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:02.914 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:02.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:03.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:03.280 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:03.280 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:03.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:03.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:03.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:03.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:03.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:03.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:03.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:03.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:03.873 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:03.873 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:04.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23589104 2023-11-02T09:26:04.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23589104 '!=' null 2023-11-02T09:26:04.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:04.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:04.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:04.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:04.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:04.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:04.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:04.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:04.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:04.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:04.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:05.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:05.108 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:05.108 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:05.108 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:05.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:05.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:05.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:05.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:05.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:05.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:05.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:05.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:05.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:06.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=35820693 2023-11-02T09:26:06.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 35820693 '!=' null 2023-11-02T09:26:06.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:06.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:06.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:06.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:06.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:06.365 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:06.365 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:06.365 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:06.365 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:06.366 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:06.366 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:06.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:06.735 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:06.736 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:06.736 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:07.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:07.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:07.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:07.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:07.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:07.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:07.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:07.304 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:07.304 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:07.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23589204 2023-11-02T09:26:07.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23589204 '!=' null 2023-11-02T09:26:07.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:07.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:07.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:08.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:08.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:08.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:08.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:08.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:08.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:08.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:08.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:08.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:08.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:08.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:08.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:09.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:09.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:09.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:09.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:09.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:09.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:09.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:09.064 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:09.065 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:09.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29705347 2023-11-02T09:26:09.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29705347 '!=' null 2023-11-02T09:26:09.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:09.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:09.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:09.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:09.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:09.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:09.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:09.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:09.861 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:09.861 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:09.861 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:10.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:10.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:10.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:10.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:10.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:10.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:10.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:10.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:10.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:10.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:10.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:10.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:10.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:11.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30578979 2023-11-02T09:26:11.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30578979 '!=' null 2023-11-02T09:26:11.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:11.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:11.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:11.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:11.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:11.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:11.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:11.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:11.543 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:11.543 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:11.543 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:11.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:11.937 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:11.937 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:11.937 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:12.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:12.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:12.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:12.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:12.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:12.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:12.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:12.446 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:12.446 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:13.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24463012 2023-11-02T09:26:13.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24463012 '!=' null 2023-11-02T09:26:13.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:13.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:13.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:13.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:13.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:13.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:13.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:13.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:13.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:13.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:13.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:13.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:13.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:13.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:13.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:14.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:14.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:14.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:14.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:14.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:14.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:14.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:14.409 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:14.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:15.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=35820719 2023-11-02T09:26:15.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 35820719 '!=' null 2023-11-02T09:26:15.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:15.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:15.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:15.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:15.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:15.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:15.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:15.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:15.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:15.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:15.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:15.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:15.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:15.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:15.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:16.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:16.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:16.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:16.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:16.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:16.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:16.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:16.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:16.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:16.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23589222 2023-11-02T09:26:16.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23589222 '!=' null 2023-11-02T09:26:16.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:16.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:16.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:17.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:17.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:17.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:17.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:17.026 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:17.026 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:17.026 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:17.026 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:17.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:17.551 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:17.552 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:17.552 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:18.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:18.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:18.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:18.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:18.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:18.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:18.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:18.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:18.214 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:18.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23589222 2023-11-02T09:26:18.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23589222 '!=' null 2023-11-02T09:26:18.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:18.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:18.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:18.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:18.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:18.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:18.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:18.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:18.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:18.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:18.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:19.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:19.231 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:19.231 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:19.231 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:20.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:20.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:20.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:20.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:20.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:20.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:20.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:20.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:20.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:20.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23589226 2023-11-02T09:26:20.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23589226 '!=' null 2023-11-02T09:26:20.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:20.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:20.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:21.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:21.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:21.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:21.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:21.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:21.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:21.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:21.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:21.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:21.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:21.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:21.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:22.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:22.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:22.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:22.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:22.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:22.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:22.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:22.081 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:22.081 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:22.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29704706 2023-11-02T09:26:22.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29704706 '!=' null 2023-11-02T09:26:22.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:22.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:22.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:22.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:22.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:22.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:22.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:22.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:22.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:22.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:22.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:23.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:23.186 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:23.186 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:23.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:23.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:23.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:23.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:23.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:23.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:23.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:23.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:23.841 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:23.841 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:24.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588880 2023-11-02T09:26:24.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588880 '!=' null 2023-11-02T09:26:24.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:24.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:24.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:24.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:24.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:24.509 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:24.509 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:24.510 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:24.510 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:24.510 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:24.510 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:25.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:25.005 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:25.005 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:25.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:25.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:25.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:25.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:25.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:25.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:25.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:25.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:25.552 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:25.552 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:26.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34946502 2023-11-02T09:26:26.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34946502 '!=' null 2023-11-02T09:26:26.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:26.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:26.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:26.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:26.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:26.309 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:26.309 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:26.310 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:26.310 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:26.310 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:26.310 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:26.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:26.691 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:26.691 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:26.691 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:27.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:27.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:27.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:27.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:27.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:27.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:27.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:27.314 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:27.314 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:28.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588957 2023-11-02T09:26:28.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588957 '!=' null 2023-11-02T09:26:28.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:28.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:28.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:28.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:28.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:28.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:28.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:28.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:28.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:28.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:28.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:28.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:28.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:28.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:28.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:29.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:29.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:29.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:29.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:29.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:29.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:29.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:29.088 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:29.088 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:30.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588778 2023-11-02T09:26:30.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588778 '!=' null 2023-11-02T09:26:30.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:30.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:30.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:30.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:30.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:30.121 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:30.121 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:30.121 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:30.121 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:30.122 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:30.122 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:30.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:30.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:30.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:30.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:31.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:31.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:31.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:31.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:31.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:31.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:31.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:31.002 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:31.002 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:31.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34946382 2023-11-02T09:26:31.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34946382 '!=' null 2023-11-02T09:26:31.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:31.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:31.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:31.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:31.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:31.706 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:31.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:31.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:31.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:31.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:31.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:32.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:32.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:32.074 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:32.074 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:32.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:32.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:32.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:32.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:32.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:32.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:32.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:32.585 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:32.585 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:33.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715123 2023-11-02T09:26:33.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715123 '!=' null 2023-11-02T09:26:33.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:33.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:33.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:33.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:33.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:33.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:33.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:33.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:33.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:33.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:33.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:33.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:33.809 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:33.810 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:33.810 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:34.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:34.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:34.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:34.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:34.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:34.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:34.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:34.541 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:34.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:35.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34946291 2023-11-02T09:26:35.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34946291 '!=' null 2023-11-02T09:26:35.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:35.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:35.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:35.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:35.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:35.589 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:35.589 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:35.589 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:35.589 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:35.589 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:35.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:35.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:35.987 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:35.987 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:35.987 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:36.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:36.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:36.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:36.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:36.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:36.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:36.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:36.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:36.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:37.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588565 2023-11-02T09:26:37.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588565 '!=' null 2023-11-02T09:26:37.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:37.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:37.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:37.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:37.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:37.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:37.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:37.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:37.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:37.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:37.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:37.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:37.744 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:37.745 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:37.745 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:38.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:38.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:38.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:38.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:38.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:38.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:38.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:38.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:38.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:39.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588577 2023-11-02T09:26:39.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588577 '!=' null 2023-11-02T09:26:39.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:39.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:39.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:39.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:39.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:39.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:39.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:39.116 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:39.116 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:39.116 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:39.116 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:39.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:39.577 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:39.577 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:39.578 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:40.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:40.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:40.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:40.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:40.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:40.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:40.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:40.313 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:40.314 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:40.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34946288 2023-11-02T09:26:40.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34946288 '!=' null 2023-11-02T09:26:40.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:40.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:40.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:41.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:41.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:41.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:41.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:41.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:41.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:41.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:41.076 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:41.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:41.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:41.515 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:41.515 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:42.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:42.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:42.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:42.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:42.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:42.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:42.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:42.185 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:42.185 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:42.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34946288 2023-11-02T09:26:42.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34946288 '!=' null 2023-11-02T09:26:42.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:42.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:42.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:42.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:42.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:42.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:42.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:42.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:42.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:42.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:42.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:43.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:43.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:43.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:43.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:43.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:43.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:43.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:43.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:43.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:43.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:43.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:43.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:43.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:44.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588689 2023-11-02T09:26:44.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588689 '!=' null 2023-11-02T09:26:44.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:44.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:44.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:44.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:44.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:44.541 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:44.541 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:44.541 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:44.541 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:44.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:44.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:44.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:44.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:44.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:44.917 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:45.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:45.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:45.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:45.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:45.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:45.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:45.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:45.736 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:45.737 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:46.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=35819925 2023-11-02T09:26:46.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 35819925 '!=' null 2023-11-02T09:26:46.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:46.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:46.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:46.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:46.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:46.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:46.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:46.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:46.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:46.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:46.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:46.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:46.867 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:46.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:46.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:47.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:47.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:47.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:47.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:47.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:47.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:47.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:47.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:47.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:48.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588718 2023-11-02T09:26:48.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588718 '!=' null 2023-11-02T09:26:48.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:48.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:48.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:48.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:48.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:48.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:48.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:48.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:48.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:48.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:48.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:48.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:48.644 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:48.645 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:48.645 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:49.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:49.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:49.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:49.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:49.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:49.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:49.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:49.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:49.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:49.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588953 2023-11-02T09:26:49.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588953 '!=' null 2023-11-02T09:26:49.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:49.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:49.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:49.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:49.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:49.894 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:49.894 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:49.894 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:49.894 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:49.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:49.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:50.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:50.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:50.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:50.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:50.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:50.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:50.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:50.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:50.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:50.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:50.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:50.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:50.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:51.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34946716 2023-11-02T09:26:51.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34946716 '!=' null 2023-11-02T09:26:51.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:51.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:51.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:51.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:51.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:51.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:51.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:51.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:51.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:51.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:51.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:52.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:52.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:52.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:52.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:52.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:52.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:52.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:52.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:52.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:52.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:52.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:52.576 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:52.577 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:53.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23589024 2023-11-02T09:26:53.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23589024 '!=' null 2023-11-02T09:26:53.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:53.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:53.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:53.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:53.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:53.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:53.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:53.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:53.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:53.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:53.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:54.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:54.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:54.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:54.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:54.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:54.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:54.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:54.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:54.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:54.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:54.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:54.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:54.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:55.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=35820415 2023-11-02T09:26:55.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 35820415 '!=' null 2023-11-02T09:26:55.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:55.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:55.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:55.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:55.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:55.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:55.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:55.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:55.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:55.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:55.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:55.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:55.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:55.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:55.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:56.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:56.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:56.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:56.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:56.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:56.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:56.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:56.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:56.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:57.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23589082 2023-11-02T09:26:57.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23589082 '!=' null 2023-11-02T09:26:57.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:57.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:57.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:57.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:57.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:57.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:57.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:57.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:57.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:57.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:57.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:57.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:57.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:57.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:57.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:26:58.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:26:58.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:26:58.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:26:58.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:26:58.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:26:58.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:26:58.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:26:58.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:26:58.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:26:59.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23589131 2023-11-02T09:26:59.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23589131 '!=' null 2023-11-02T09:26:59.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:26:59.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:26:59.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:26:59.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:26:59.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:26:59.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:26:59.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:26:59.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:26:59.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:26:59.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:26:59.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:26:59.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:26:59.577 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:26:59.577 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:26:59.577 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:00.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:00.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:00.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:00.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:00.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:00.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:00.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:00.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:00.473 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:01.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=35820463 2023-11-02T09:27:01.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 35820463 '!=' null 2023-11-02T09:27:01.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:01.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:01.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:01.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:01.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:01.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:01.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:01.343 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:01.343 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:01.343 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:01.343 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:01.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:01.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:01.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:01.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:02.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:02.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:02.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:02.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:02.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:02.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:02.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:02.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:02.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:02.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=35820463 2023-11-02T09:27:02.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 35820463 '!=' null 2023-11-02T09:27:02.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:02.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:02.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:03.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:03.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:03.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:03.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:03.079 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:03.079 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:03.079 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:03.079 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:03.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:03.480 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:03.481 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:03.481 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:04.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:04.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:04.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:04.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:04.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:04.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:04.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:04.190 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:04.190 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:04.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462522 2023-11-02T09:27:04.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462522 '!=' null 2023-11-02T09:27:04.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:04.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:04.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:04.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:04.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:04.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:04.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:04.893 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:04.893 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:04.893 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:04.893 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:05.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:05.270 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:05.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:05.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:05.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:05.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:05.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:05.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:05.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:05.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:05.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:05.815 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:05.815 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:06.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=36693740 2023-11-02T09:27:06.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 36693740 '!=' null 2023-11-02T09:27:06.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:06.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:06.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:06.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:06.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:06.630 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:06.630 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:06.630 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:06.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:06.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:06.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:07.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:07.002 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:07.003 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:07.003 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:07.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:07.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:07.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:07.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:07.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:07.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:07.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:07.510 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:07.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:08.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462456 2023-11-02T09:27:08.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462456 '!=' null 2023-11-02T09:27:08.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:08.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:08.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:08.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:08.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:08.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:08.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:08.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:08.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:08.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:08.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:08.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:08.637 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:08.637 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:08.637 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:09.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:09.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:09.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:09.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:09.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:09.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:09.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:09.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:09.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:10.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462418 2023-11-02T09:27:10.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462418 '!=' null 2023-11-02T09:27:10.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:10.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:10.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:10.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:10.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:10.116 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:10.116 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:10.116 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:10.116 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:10.116 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:10.117 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:10.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:10.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:10.597 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:10.597 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:11.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:11.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:11.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:11.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:11.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:11.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:11.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:11.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:11.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:11.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34946724 2023-11-02T09:27:11.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34946724 '!=' null 2023-11-02T09:27:11.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:11.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:11.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:11.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:11.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:11.767 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:11.767 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:11.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:11.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:11.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:11.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:12.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:12.134 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:12.134 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:12.135 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:12.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:12.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:12.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:12.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:12.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:12.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:12.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:12.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:12.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:13.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715282 2023-11-02T09:27:13.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715282 '!=' null 2023-11-02T09:27:13.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:13.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:13.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:13.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:13.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:13.507 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:13.507 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:13.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:13.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:13.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:13.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:13.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:13.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:13.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:13.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:14.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:14.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:14.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:14.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:14.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:14.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:14.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:14.403 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:14.404 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:15.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34946771 2023-11-02T09:27:15.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34946771 '!=' null 2023-11-02T09:27:15.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:15.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:15.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:15.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:15.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:15.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:15.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:15.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:15.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:15.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:15.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:15.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:15.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:15.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:15.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:16.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:16.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:16.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:16.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:16.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:16.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:16.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:16.256 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:16.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:16.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715105 2023-11-02T09:27:16.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715105 '!=' null 2023-11-02T09:27:16.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:16.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:16.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:17.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:17.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:17.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:17.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:17.086 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:17.086 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:17.086 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:17.086 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:17.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:17.456 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:17.456 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:17.456 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:18.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:18.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:18.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:18.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:18.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:18.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:18.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:18.067 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:18.067 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:18.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715105 2023-11-02T09:27:18.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715105 '!=' null 2023-11-02T09:27:18.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:18.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:18.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:18.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:18.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:18.738 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:18.739 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:18.739 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:18.739 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:18.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:18.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:19.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:19.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:19.114 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:19.114 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:19.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:19.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:19.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:19.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:19.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:19.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:19.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:19.705 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:19.705 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:20.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715183 2023-11-02T09:27:20.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715183 '!=' null 2023-11-02T09:27:20.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:20.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:20.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:20.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:20.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:20.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:20.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:20.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:20.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:20.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:20.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:20.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:20.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:20.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:20.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:21.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:21.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:21.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:21.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:21.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:21.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:21.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:21.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:21.540 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:22.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34946414 2023-11-02T09:27:22.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34946414 '!=' null 2023-11-02T09:27:22.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:22.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:22.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:22.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:22.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:22.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:22.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:22.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:22.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:22.244 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:22.244 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:22.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:22.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:22.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:22.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:23.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:23.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:23.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:23.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:23.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:23.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:23.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:23.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:23.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:23.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462360 2023-11-02T09:27:23.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462360 '!=' null 2023-11-02T09:27:23.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:23.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:23.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:23.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:23.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:23.847 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:23.847 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:23.848 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:23.848 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:23.848 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:23.848 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:24.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:24.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:24.217 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:24.217 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:24.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:24.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:24.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:24.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:24.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:24.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:24.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:24.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:24.731 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:25.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34946547 2023-11-02T09:27:25.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34946547 '!=' null 2023-11-02T09:27:25.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:25.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:25.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:25.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:25.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:25.578 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:25.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:25.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:25.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:25.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:25.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:26.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:26.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:26.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:26.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:26.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:26.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:26.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:26.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:26.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:26.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:26.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:26.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:26.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:27.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715168 2023-11-02T09:27:27.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715168 '!=' null 2023-11-02T09:27:27.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:27.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:27.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:27.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:27.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:27.560 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:27.560 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:27.561 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:27.561 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:27.561 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:27.561 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:27.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:27.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:27.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:27.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:28.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:28.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:28.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:28.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:28.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:28.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:28.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:28.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:28.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:29.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715511 2023-11-02T09:27:29.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715511 '!=' null 2023-11-02T09:27:29.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:29.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:29.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:29.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:29.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:29.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:29.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:29.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:29.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:29.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:29.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:29.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:29.584 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:29.585 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:29.585 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:30.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:30.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:30.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:30.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:30.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:30.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:30.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:30.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:30.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:31.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34946818 2023-11-02T09:27:31.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34946818 '!=' null 2023-11-02T09:27:31.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:31.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:31.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:31.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:31.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:31.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:31.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:31.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:31.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:31.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:31.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:31.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:31.660 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:31.660 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:31.661 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:32.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:32.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:32.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:32.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:32.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:32.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:32.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:32.292 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:32.292 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:32.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715148 2023-11-02T09:27:32.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715148 '!=' null 2023-11-02T09:27:32.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:32.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:32.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:33.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:33.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:33.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:33.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:33.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:33.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:33.054 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:33.054 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:33.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:33.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:33.592 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:33.592 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:34.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:34.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:34.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:34.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:34.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:34.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:34.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:34.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:34.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:35.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34946356 2023-11-02T09:27:35.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34946356 '!=' null 2023-11-02T09:27:35.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:35.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:35.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:35.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:35.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:35.163 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:35.163 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:35.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:35.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:35.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:35.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:35.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:35.532 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:35.532 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:35.532 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:36.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:36.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:36.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:36.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:36.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:36.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:36.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:36.253 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:36.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:37.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=36693464 2023-11-02T09:27:37.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 36693464 '!=' null 2023-11-02T09:27:37.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:37.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:37.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:37.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:37.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:37.251 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:37.251 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:37.251 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:37.252 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:37.252 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:37.252 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:37.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:37.620 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:37.620 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:37.620 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:38.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:38.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:38.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:38.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:38.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:38.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:38.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:38.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:38.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:38.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18346805 2023-11-02T09:27:38.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18346805 '!=' null 2023-11-02T09:27:38.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:38.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:38.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:38.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:38.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:38.882 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:38.882 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:38.882 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:38.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:38.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:38.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:39.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:27:39.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:39.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:39.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:39.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2023-11-02T09:27:39.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:27:39.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:39.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:39.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:39.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:39.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:39.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:39.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:40.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18346805 2023-11-02T09:27:40.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18346805 '!=' null 2023-11-02T09:27:40.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:40.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:40.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:40.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:40.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:40.578 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:40.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:40.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:40.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:40.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:40.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:41.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:27:41.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:41.139 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:41.139 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:41.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2023-11-02T09:27:41.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 0 2023-11-02T09:27:41.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2023-11-02T09:27:41.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2023-11-02T09:27:41.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:41.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:41.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:41.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:41.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:41.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:41.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:41.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:41.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:41.792 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:42.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:27:42.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:42.261 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:42.261 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:42.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2023-11-02T09:27:42.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:27:42.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:42.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:42.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:42.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:42.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:42.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:42.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:43.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13105085 2023-11-02T09:27:43.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13105085 '!=' null 2023-11-02T09:27:43.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:43.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:43.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:43.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:43.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:43.480 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:43.481 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:43.481 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:43.481 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:43.482 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:43.482 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:43.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:27:43.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:43.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:43.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:44.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2023-11-02T09:27:44.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:27:44.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:44.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:44.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:44.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:44.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:44.545 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:44.545 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:45.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210233 2023-11-02T09:27:45.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210233 '!=' null 2023-11-02T09:27:45.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:45.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:45.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:45.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:45.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:45.207 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:45.207 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:45.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:45.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:45.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:45.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:45.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:27:45.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:45.583 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:45.583 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:46.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2023-11-02T09:27:46.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:27:46.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:46.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:46.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:46.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:46.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:46.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:46.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:46.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210233 2023-11-02T09:27:46.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210233 '!=' null 2023-11-02T09:27:46.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:46.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:46.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:46.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:46.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:46.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:46.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:46.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:46.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:46.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:46.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:47.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:27:47.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:47.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:47.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:47.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2023-11-02T09:27:47.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:27:47.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:47.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:47.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:47.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:47.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:47.635 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:47.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:48.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210286 2023-11-02T09:27:48.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210286 '!=' null 2023-11-02T09:27:48.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:48.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:48.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:48.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:48.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:48.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:48.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:48.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:48.246 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:48.246 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:48.246 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:48.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:27:48.607 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:48.607 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:48.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:49.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2023-11-02T09:27:49.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:27:49.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:49.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:49.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:49.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:49.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:49.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:49.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:49.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13105125 2023-11-02T09:27:49.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13105125 '!=' null 2023-11-02T09:27:49.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:49.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:49.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:49.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:49.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:49.762 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:49.763 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:49.763 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:49.763 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:49.764 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:49.764 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:50.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:27:50.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:50.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:50.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:50.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2023-11-02T09:27:50.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:27:50.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:27:50.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:27:50.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:27:50.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:27:50.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:27:50.630 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:27:50.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:27:51.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13105206 2023-11-02T09:27:51.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13105206 '!=' null 2023-11-02T09:27:51.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:27:51.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-11-02T09:27:51.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:27:51.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:27:51.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:27:51.263 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:27:51.263 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:27:51.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:27:51.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:27:51.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:27:51.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:27:51.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:27:51.625 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:27:51.625 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:27:51.625 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:27:52.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2023-11-02T09:27:52.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T09:27:52.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T09:27:52.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:525: TEST_recovery_pool_priority: '[' 0 '!=' 0 ']' 2023-11-02T09:27:52.129 INFO:tasks.workunit.client.0.smithi116.stdout:TEST PASSED 2023-11-02T09:27:52.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:529: TEST_recovery_pool_priority: echo TEST PASSED 2023-11-02T09:27:52.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:532: TEST_recovery_pool_priority: delete_pool test1 2023-11-02T09:27:52.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2023-11-02T09:27:52.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2023-11-02T09:27:52.644 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test1' does not exist 2023-11-02T09:27:52.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:533: TEST_recovery_pool_priority: delete_pool test2 2023-11-02T09:27:52.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2023-11-02T09:27:52.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2023-11-02T09:27:53.318 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test2' does not exist 2023-11-02T09:27:53.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:534: TEST_recovery_pool_priority: kill_daemons td/osd-recovery-prio 2023-11-02T09:27:53.330 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:27:53.330 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:27:53.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:27:53.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:27:53.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:27:58.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:27:58.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:535: TEST_recovery_pool_priority: return 0 2023-11-02T09:27:58.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:41: run: teardown td/osd-recovery-prio 2023-11-02T09:27:58.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2023-11-02T09:27:58.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:27:58.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2023-11-02T09:27:58.644 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:27:58.645 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:27:58.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:27:58.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:27:58.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:27:58.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:27:58.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:27:58.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:27:58.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:27:58.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:27:58.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:27:58.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:27:58.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:27:58.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:27:58.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:27:58.653 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:27:58.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:27:58.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:27:58.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2023-11-02T09:27:58.691 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:27:58.691 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:27:58.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:27:58.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.261297 2023-11-02T09:27:58.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:27:58.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:27:58.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: for func in $funcs 2023-11-02T09:27:58.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:39: run: setup td/osd-recovery-prio 2023-11-02T09:27:58.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2023-11-02T09:27:58.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2023-11-02T09:27:58.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2023-11-02T09:27:58.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:27:58.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2023-11-02T09:27:58.696 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:27:58.696 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:27:58.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:27:58.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:27:58.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:27:58.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:27:58.699 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:27:58.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:27:58.699 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:27:58.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:27:58.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:27:58.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:27:58.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:27:58.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:27:58.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:27:58.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:27:58.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:27:58.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:27:58.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2023-11-02T09:27:58.703 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:27:58.703 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:27:58.704 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:27:58.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.261297 2023-11-02T09:27:58.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:27:58.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:27:58.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2023-11-02T09:27:58.706 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T09:27:58.706 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:27:58.706 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:27:58.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.261297 2023-11-02T09:27:58.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T09:27:58.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-11-02T09:27:58.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T09:27:58.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2023-11-02T09:27:58.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:40: run: TEST_recovery_priority td/osd-recovery-prio 2023-11-02T09:27:58.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:47: TEST_recovery_priority: local dir=td/osd-recovery-prio 2023-11-02T09:27:58.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:48: TEST_recovery_priority: local pools=10 2023-11-02T09:27:58.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:49: TEST_recovery_priority: local OSDS=5 2023-11-02T09:27:58.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:50: TEST_recovery_priority: local max_tries=10 2023-11-02T09:27:58.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:52: TEST_recovery_priority: run_mon td/osd-recovery-prio a 2023-11-02T09:27:58.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2023-11-02T09:27:58.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T09:27:58.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T09:27:58.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T09:27:58.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2023-11-02T09:27:58.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2023-11-02T09:27:59.121 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T09:27:59.121 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:27:59.121 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:27:59.121 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:27:59.122 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:27:59.122 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:27:59.122 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.261297/$cluster-$name.asok' 2023-11-02T09:27:59.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T09:27:59.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T09:27:59.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T09:27:59.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:27:59.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:27:59.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T09:27:59.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T09:27:59.324 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:27:59.324 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:27:59.324 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:27:59.325 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:27:59.325 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:27:59.325 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:27:59.326 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.261297/ceph-mon.a.asok 2023-11-02T09:27:59.326 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:27:59.326 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.261297/ceph-mon.a.asok config get fsid 2023-11-02T09:27:59.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T09:27:59.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:27:59.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:27:59.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T09:27:59.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T09:27:59.412 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:27:59.413 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:27:59.413 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:27:59.413 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:27:59.413 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:27:59.413 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:27:59.414 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.261297/ceph-mon.a.asok 2023-11-02T09:27:59.414 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:27:59.414 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.261297/ceph-mon.a.asok config get mon_host 2023-11-02T09:27:59.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: run_mgr td/osd-recovery-prio x 2023-11-02T09:27:59.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2023-11-02T09:27:59.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T09:27:59.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T09:27:59.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T09:27:59.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2023-11-02T09:27:59.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T09:28:00.343 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T09:28:00.344 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:28:00.344 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:28:00.344 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:28:00.344 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:28:00.344 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:28:00.345 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.261297/$cluster-$name.asok' 2023-11-02T09:28:00.345 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:28:00.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:28:00.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:54: TEST_recovery_priority: export CEPH_ARGS 2023-11-02T09:28:00.372 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: expr 5 - 1 2023-11-02T09:28:00.373 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: seq 0 4 2023-11-02T09:28:00.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-11-02T09:28:00.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 0 2023-11-02T09:28:00.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2023-11-02T09:28:00.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:28:00.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T09:28:00.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:28:00.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2023-11-02T09:28:00.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2023-11-02T09:28:00.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:28:00.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:28:00.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:28:00.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2023-11-02T09:28:00.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2023-11-02T09:28:00.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:28:00.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:28:00.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2023-11-02T09:28:00.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:28:00.381 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:28:00.381 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:28:00.381 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:28:00.381 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:28:00.381 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:28:00.382 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.261297/$cluster-$name.asok' 2023-11-02T09:28:00.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' 2023-11-02T09:28:00.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:28:00.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:28:00.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:28:00.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2023-11-02T09:28:00.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2023-11-02T09:28:00.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:28:00.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:28:00.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:28:00.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:28:00.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:28:00.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:28:00.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/0 2023-11-02T09:28:00.385 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:28:00.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b6d9b064-ed78-476e-a70c-243c8fdb77cc 2023-11-02T09:28:00.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b6d9b064-ed78-476e-a70c-243c8fdb77cc' 2023-11-02T09:28:00.386 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 b6d9b064-ed78-476e-a70c-243c8fdb77cc 2023-11-02T09:28:00.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:28:00.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCga0Nloo95FxAApa5AfeIAPSy5Zzc1JylXMQ== 2023-11-02T09:28:00.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCga0Nloo95FxAApa5AfeIAPSy5Zzc1JylXMQ=="}' 2023-11-02T09:28:00.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b6d9b064-ed78-476e-a70c-243c8fdb77cc -i td/osd-recovery-prio/0/new.json 2023-11-02T09:28:00.738 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:28:00.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/0/new.json 2023-11-02T09:28:00.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCga0Nloo95FxAApa5AfeIAPSy5Zzc1JylXMQ== --osd-uuid b6d9b064-ed78-476e-a70c-243c8fdb77cc 2023-11-02T09:28:00.782 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:00.779+0000 7fe631449500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:00.784 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:00.781+0000 7fe631449500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:00.785 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:00.783+0000 7fe631449500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:00.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:00.783+0000 7fe631449500 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2023-11-02T09:28:00.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:00.783+0000 7fe631449500 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2023-11-02T09:28:00.786 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:00.783+0000 7fe631449500 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2023-11-02T09:28:00.862 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:00.860+0000 7fe631449500 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2023-11-02T09:28:06.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2023-11-02T09:28:06.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:28:06.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T09:28:06.109 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T09:28:06.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:28:06.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T09:28:06.653 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T09:28:06.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:28:06.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:28:06.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:28:06.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:28:06.681 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:06.678+0000 7fb8d720c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:06.689 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:06.686+0000 7fb8d720c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:06.690 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:06.687+0000 7fb8d720c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:07.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T09:28:07.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:28:07.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T09:28:07.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:28:07.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:28:07.025 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:28:07.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:28:07.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:28:07.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:28:07.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:28:07.209 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:07.207+0000 7fb8d720c500 -1 Falling back to public interface 2023-11-02T09:28:07.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:28:08.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:28:08.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:28:08.437 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:28:08.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:28:08.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:28:08.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:28:08.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:28:09.532 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:09.529+0000 7fb8d720c500 -1 osd.0 0 log_to_monitors true 2023-11-02T09:28:09.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:28:09.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:28:09.855 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:28:09.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:28:09.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:28:09.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:28:10.350 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/281212,v1:127.0.0.1:6803/281212] [v2:127.0.0.1:6804/281212,v1:127.0.0.1:6805/281212] exists,up b6d9b064-ed78-476e-a70c-243c8fdb77cc 2023-11-02T09:28:10.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:28:10.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:28:10.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:28:10.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-11-02T09:28:10.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 1 2023-11-02T09:28:10.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2023-11-02T09:28:10.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:28:10.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-11-02T09:28:10.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:28:10.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2023-11-02T09:28:10.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2023-11-02T09:28:10.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:28:10.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:28:10.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:28:10.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2023-11-02T09:28:10.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2023-11-02T09:28:10.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:28:10.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:28:10.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2023-11-02T09:28:10.358 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:28:10.358 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:28:10.358 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:28:10.358 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:28:10.358 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:28:10.358 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:28:10.359 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.261297/$cluster-$name.asok' 2023-11-02T09:28:10.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' 2023-11-02T09:28:10.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:28:10.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:28:10.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:28:10.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2023-11-02T09:28:10.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2023-11-02T09:28:10.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:28:10.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:28:10.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:28:10.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:28:10.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:28:10.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:28:10.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/1 2023-11-02T09:28:10.361 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:28:10.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=730f00a8-0d34-497d-ad0c-83927159707d 2023-11-02T09:28:10.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 730f00a8-0d34-497d-ad0c-83927159707d' 2023-11-02T09:28:10.362 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 730f00a8-0d34-497d-ad0c-83927159707d 2023-11-02T09:28:10.362 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:28:10.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCqa0NlIS0CFhAAUKGscmojrtxzACsqCoTKgg== 2023-11-02T09:28:10.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCqa0NlIS0CFhAAUKGscmojrtxzACsqCoTKgg=="}' 2023-11-02T09:28:10.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 730f00a8-0d34-497d-ad0c-83927159707d -i td/osd-recovery-prio/1/new.json 2023-11-02T09:28:10.966 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:28:10.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/1/new.json 2023-11-02T09:28:10.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCqa0NlIS0CFhAAUKGscmojrtxzACsqCoTKgg== --osd-uuid 730f00a8-0d34-497d-ad0c-83927159707d 2023-11-02T09:28:11.007 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:11.004+0000 7eff39307500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:11.026 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:11.023+0000 7eff39307500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:11.026 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:11.024+0000 7eff39307500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:11.027 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:11.024+0000 7eff39307500 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2023-11-02T09:28:11.027 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:11.024+0000 7eff39307500 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2023-11-02T09:28:11.028 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:11.024+0000 7eff39307500 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2023-11-02T09:28:11.134 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:11.131+0000 7eff39307500 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2023-11-02T09:28:16.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2023-11-02T09:28:16.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:28:16.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-11-02T09:28:16.549 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2023-11-02T09:28:16.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:28:17.064 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T09:28:17.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-11-02T09:28:17.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:28:17.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:28:17.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:28:17.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:28:17.092 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:17.089+0000 7f063840d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:17.122 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:17.119+0000 7f063840d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:17.123 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:17.120+0000 7f063840d500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:17.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-11-02T09:28:17.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:28:17.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T09:28:17.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:28:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:28:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:28:17.441 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:28:17.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:28:17.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:28:17.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:28:17.658 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:17.655+0000 7f063840d500 -1 Falling back to public interface 2023-11-02T09:28:17.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:28:18.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:28:18.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:28:18.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:28:18.854 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:28:18.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:28:18.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:28:19.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:28:19.916 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:19.913+0000 7f063840d500 -1 osd.1 0 log_to_monitors true 2023-11-02T09:28:20.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:28:20.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:28:20.273 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:28:20.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:28:20.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:28:20.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:28:20.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:28:21.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:28:21.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:28:21.665 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:28:21.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:28:21.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:28:21.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:28:22.103 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/282570,v1:127.0.0.1:6811/282570] [v2:127.0.0.1:6812/282570,v1:127.0.0.1:6813/282570] exists,up 730f00a8-0d34-497d-ad0c-83927159707d 2023-11-02T09:28:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:28:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:28:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:28:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-11-02T09:28:22.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 2 2023-11-02T09:28:22.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2023-11-02T09:28:22.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:28:22.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-11-02T09:28:22.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:28:22.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/2 2023-11-02T09:28:22.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2023-11-02T09:28:22.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:28:22.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:28:22.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:28:22.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/2' 2023-11-02T09:28:22.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/2/journal' 2023-11-02T09:28:22.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:28:22.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:28:22.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2023-11-02T09:28:22.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:28:22.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:28:22.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:28:22.112 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:28:22.113 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:28:22.113 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:28:22.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.261297/$cluster-$name.asok' 2023-11-02T09:28:22.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' 2023-11-02T09:28:22.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:28:22.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:28:22.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:28:22.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2023-11-02T09:28:22.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2023-11-02T09:28:22.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:28:22.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:28:22.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:28:22.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:28:22.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:28:22.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:28:22.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/2 2023-11-02T09:28:22.116 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:28:22.116 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 77775c29-b078-4177-98cc-30d2c60c0708 2023-11-02T09:28:22.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=77775c29-b078-4177-98cc-30d2c60c0708 2023-11-02T09:28:22.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 77775c29-b078-4177-98cc-30d2c60c0708' 2023-11-02T09:28:22.117 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:28:22.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC2a0NlwLtXBxAANeMbAINm35jzIo3NOJEyqg== 2023-11-02T09:28:22.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC2a0NlwLtXBxAANeMbAINm35jzIo3NOJEyqg=="}' 2023-11-02T09:28:22.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 77775c29-b078-4177-98cc-30d2c60c0708 -i td/osd-recovery-prio/2/new.json 2023-11-02T09:28:22.697 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:28:22.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/2/new.json 2023-11-02T09:28:22.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC2a0NlwLtXBxAANeMbAINm35jzIo3NOJEyqg== --osd-uuid 77775c29-b078-4177-98cc-30d2c60c0708 2023-11-02T09:28:22.738 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:22.735+0000 7f0d65b9b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:22.739 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:22.737+0000 7f0d65b9b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:22.740 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:22.738+0000 7f0d65b9b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:22.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:22.738+0000 7f0d65b9b500 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2023-11-02T09:28:22.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:22.738+0000 7f0d65b9b500 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2023-11-02T09:28:22.742 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:22.738+0000 7f0d65b9b500 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2023-11-02T09:28:22.787 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:22.785+0000 7f0d65b9b500 -1 bluestore(td/osd-recovery-prio/2) _read_fsid unparsable uuid 2023-11-02T09:28:28.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/2/keyring 2023-11-02T09:28:28.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:28:28.524 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2023-11-02T09:28:28.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-11-02T09:28:28.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:28:29.037 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2023-11-02T09:28:29.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-11-02T09:28:29.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:28:29.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:28:29.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:28:29.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:28:29.065 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:29.061+0000 7f2f178da500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:29.086 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:29.082+0000 7f2f178da500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:29.087 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:29.083+0000 7f2f178da500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:29.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-11-02T09:28:29.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:28:29.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-11-02T09:28:29.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:28:29.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:28:29.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:28:29.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:28:29.492 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:28:29.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:28:29.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:28:29.623 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:29.619+0000 7f2f178da500 -1 Falling back to public interface 2023-11-02T09:28:29.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:28:30.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:28:30.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:28:30.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:28:30.870 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:28:30.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:28:30.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:28:31.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:28:31.905 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:31.902+0000 7f2f178da500 -1 osd.2 0 log_to_monitors true 2023-11-02T09:28:32.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:28:32.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:28:32.295 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:28:32.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:28:32.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:28:32.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-11-02T09:28:32.799 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/283953,v1:127.0.0.1:6819/283953] [v2:127.0.0.1:6820/283953,v1:127.0.0.1:6821/283953] exists,up 77775c29-b078-4177-98cc-30d2c60c0708 2023-11-02T09:28:32.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:28:32.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:28:32.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:28:32.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-11-02T09:28:32.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 3 2023-11-02T09:28:32.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2023-11-02T09:28:32.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:28:32.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2023-11-02T09:28:32.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:28:32.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/3 2023-11-02T09:28:32.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2023-11-02T09:28:32.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:28:32.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:28:32.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:28:32.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/3' 2023-11-02T09:28:32.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/3/journal' 2023-11-02T09:28:32.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:28:32.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:28:32.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2023-11-02T09:28:32.807 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:28:32.807 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:28:32.807 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:28:32.808 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:28:32.808 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:28:32.808 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:28:32.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.261297/$cluster-$name.asok' 2023-11-02T09:28:32.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' 2023-11-02T09:28:32.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:28:32.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:28:32.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:28:32.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2023-11-02T09:28:32.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2023-11-02T09:28:32.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:28:32.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:28:32.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:28:32.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:28:32.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:28:32.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:28:32.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/3 2023-11-02T09:28:32.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:28:32.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eab8b84b-c18c-4be9-8d5e-96f8418a8e67 2023-11-02T09:28:32.812 INFO:tasks.workunit.client.0.smithi116.stdout:add osd3 eab8b84b-c18c-4be9-8d5e-96f8418a8e67 2023-11-02T09:28:32.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 eab8b84b-c18c-4be9-8d5e-96f8418a8e67' 2023-11-02T09:28:32.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:28:32.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDAa0Nlr7XVMBAAjf7oGviWR8S6hjv+V4e/kQ== 2023-11-02T09:28:32.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDAa0Nlr7XVMBAAjf7oGviWR8S6hjv+V4e/kQ=="}' 2023-11-02T09:28:32.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eab8b84b-c18c-4be9-8d5e-96f8418a8e67 -i td/osd-recovery-prio/3/new.json 2023-11-02T09:28:33.384 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:28:33.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/3/new.json 2023-11-02T09:28:33.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDAa0Nlr7XVMBAAjf7oGviWR8S6hjv+V4e/kQ== --osd-uuid eab8b84b-c18c-4be9-8d5e-96f8418a8e67 2023-11-02T09:28:33.426 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:33.423+0000 7fee1281c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:33.427 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:33.425+0000 7fee1281c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:33.428 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:33.426+0000 7fee1281c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:33.429 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:33.426+0000 7fee1281c500 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2023-11-02T09:28:33.429 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:33.426+0000 7fee1281c500 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2023-11-02T09:28:33.429 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:33.426+0000 7fee1281c500 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2023-11-02T09:28:33.466 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:33.464+0000 7fee1281c500 -1 bluestore(td/osd-recovery-prio/3) _read_fsid unparsable uuid 2023-11-02T09:28:39.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/3/keyring 2023-11-02T09:28:39.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:28:39.555 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd3 key to auth repository 2023-11-02T09:28:39.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2023-11-02T09:28:39.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:28:40.065 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.3 2023-11-02T09:28:40.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2023-11-02T09:28:40.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:28:40.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:28:40.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:28:40.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:28:40.093 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:40.091+0000 7f60160fa500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:40.115 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:40.112+0000 7f60160fa500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:40.115 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:40.113+0000 7f60160fa500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:40.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2023-11-02T09:28:40.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:28:40.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2023-11-02T09:28:40.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:28:40.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:28:40.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:28:40.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:28:40.521 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:28:40.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:28:40.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T09:28:40.644 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:40.641+0000 7f60160fa500 -1 Falling back to public interface 2023-11-02T09:28:40.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:28:41.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:28:41.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:28:41.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:28:41.939 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:28:41.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:28:41.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T09:28:42.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:28:42.916 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:42.913+0000 7f60160fa500 -1 osd.3 0 log_to_monitors true 2023-11-02T09:28:43.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:28:43.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:28:43.365 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:28:43.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:28:43.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:28:43.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-11-02T09:28:43.777 INFO:tasks.workunit.client.0.smithi116.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/285311,v1:127.0.0.1:6827/285311] [v2:127.0.0.1:6828/285311,v1:127.0.0.1:6829/285311] exists,up eab8b84b-c18c-4be9-8d5e-96f8418a8e67 2023-11-02T09:28:43.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:28:43.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:28:43.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:28:43.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-11-02T09:28:43.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 4 2023-11-02T09:28:43.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2023-11-02T09:28:43.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:28:43.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2023-11-02T09:28:43.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:28:43.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/4 2023-11-02T09:28:43.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2023-11-02T09:28:43.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:28:43.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:28:43.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:28:43.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/4' 2023-11-02T09:28:43.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/4/journal' 2023-11-02T09:28:43.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:28:43.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:28:43.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2023-11-02T09:28:43.785 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:28:43.785 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:28:43.785 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:28:43.785 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:28:43.785 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:28:43.785 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:28:43.786 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.261297/$cluster-$name.asok' 2023-11-02T09:28:43.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' 2023-11-02T09:28:43.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:28:43.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:28:43.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:28:43.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2023-11-02T09:28:43.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2023-11-02T09:28:43.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:28:43.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:28:43.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:28:43.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:28:43.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:28:43.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:28:43.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/4 2023-11-02T09:28:43.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:28:43.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7a731143-3c78-4ac1-b31a-0e8258ccdd25 2023-11-02T09:28:43.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 7a731143-3c78-4ac1-b31a-0e8258ccdd25' 2023-11-02T09:28:43.789 INFO:tasks.workunit.client.0.smithi116.stdout:add osd4 7a731143-3c78-4ac1-b31a-0e8258ccdd25 2023-11-02T09:28:43.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:28:43.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDLa0NlWEpyLxAAZ4o93MNJzxY0kLHlyNcyfQ== 2023-11-02T09:28:43.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDLa0NlWEpyLxAAZ4o93MNJzxY0kLHlyNcyfQ=="}' 2023-11-02T09:28:43.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7a731143-3c78-4ac1-b31a-0e8258ccdd25 -i td/osd-recovery-prio/4/new.json 2023-11-02T09:28:44.503 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:28:44.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/4/new.json 2023-11-02T09:28:44.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDLa0NlWEpyLxAAZ4o93MNJzxY0kLHlyNcyfQ== --osd-uuid 7a731143-3c78-4ac1-b31a-0e8258ccdd25 2023-11-02T09:28:44.544 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:44.541+0000 7f40c6a25500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:44.553 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:44.551+0000 7f40c6a25500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:44.554 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:44.552+0000 7f40c6a25500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:44.555 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:44.552+0000 7f40c6a25500 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2023-11-02T09:28:44.555 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:44.552+0000 7f40c6a25500 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2023-11-02T09:28:44.555 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:44.552+0000 7f40c6a25500 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2023-11-02T09:28:44.645 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:44.642+0000 7f40c6a25500 -1 bluestore(td/osd-recovery-prio/4) _read_fsid unparsable uuid 2023-11-02T09:28:50.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/4/keyring 2023-11-02T09:28:50.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:28:50.418 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd4 key to auth repository 2023-11-02T09:28:50.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2023-11-02T09:28:50.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:28:50.950 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.4 2023-11-02T09:28:50.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2023-11-02T09:28:50.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=b514020f-f106-4978-878e-bcb446f7d9a1 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.261297/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:28:50.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:28:50.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:28:50.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:28:50.978 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:50.975+0000 7fc85b82b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:51.011 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:51.008+0000 7fc85b82b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:51.012 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:51.009+0000 7fc85b82b500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:28:51.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2023-11-02T09:28:51.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:28:51.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2023-11-02T09:28:51.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:28:51.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:28:51.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:28:51.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:28:51.326 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:28:51.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:28:51.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2023-11-02T09:28:51.543 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:51.541+0000 7fc85b82b500 -1 Falling back to public interface 2023-11-02T09:28:51.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:28:52.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:28:52.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:28:52.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:28:52.704 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:28:52.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:28:52.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2023-11-02T09:28:53.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:28:53.788 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:28:53.785+0000 7fc85b82b500 -1 osd.4 0 log_to_monitors true 2023-11-02T09:28:54.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:28:54.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:28:54.128 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:28:54.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:28:54.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:28:54.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2023-11-02T09:28:54.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:28:55.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:28:55.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:28:55.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:28:55.615 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:28:55.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:28:55.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2023-11-02T09:28:56.019 INFO:tasks.workunit.client.0.smithi116.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/286669,v1:127.0.0.1:6835/286669] [v2:127.0.0.1:6836/286669,v1:127.0.0.1:6837/286669] exists,up 7a731143-3c78-4ac1-b31a-0e8258ccdd25 2023-11-02T09:28:56.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:28:56.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:28:56.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:28:56.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: seq 1 10 2023-11-02T09:28:56.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:28:56.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test1 1 1 2023-11-02T09:28:56.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2023-11-02T09:28:56.747 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test1' already exists 2023-11-02T09:28:56.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:28:57.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test1 size 2 2023-11-02T09:28:58.780 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 size to 2 2023-11-02T09:28:58.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:28:58.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test2 1 1 2023-11-02T09:28:58.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2023-11-02T09:28:59.361 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test2' already exists 2023-11-02T09:28:59.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:29:00.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test2 size 2 2023-11-02T09:29:01.245 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 2 size to 2 2023-11-02T09:29:01.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:01.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test3 1 1 2023-11-02T09:29:01.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2023-11-02T09:29:02.022 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test3' already exists 2023-11-02T09:29:02.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:29:03.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test3 size 2 2023-11-02T09:29:04.083 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 3 size to 2 2023-11-02T09:29:04.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:04.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test4 1 1 2023-11-02T09:29:04.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test4 1 1 2023-11-02T09:29:04.744 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test4' already exists 2023-11-02T09:29:04.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:29:05.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test4 size 2 2023-11-02T09:29:06.832 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 4 size to 2 2023-11-02T09:29:06.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:06.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test5 1 1 2023-11-02T09:29:06.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test5 1 1 2023-11-02T09:29:07.464 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test5' already exists 2023-11-02T09:29:07.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:29:08.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test5 size 2 2023-11-02T09:29:09.338 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 5 size to 2 2023-11-02T09:29:09.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:09.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test6 1 1 2023-11-02T09:29:09.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test6 1 1 2023-11-02T09:29:10.021 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test6' already exists 2023-11-02T09:29:10.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:29:11.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test6 size 2 2023-11-02T09:29:11.920 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 6 size to 2 2023-11-02T09:29:11.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:11.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test7 1 1 2023-11-02T09:29:11.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test7 1 1 2023-11-02T09:29:12.640 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test7' already exists 2023-11-02T09:29:12.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:29:13.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test7 size 2 2023-11-02T09:29:14.709 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 7 size to 2 2023-11-02T09:29:14.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:14.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test8 1 1 2023-11-02T09:29:14.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test8 1 1 2023-11-02T09:29:15.443 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test8' already exists 2023-11-02T09:29:15.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:29:16.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test8 size 2 2023-11-02T09:29:17.407 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 8 size to 2 2023-11-02T09:29:17.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:17.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test9 1 1 2023-11-02T09:29:17.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test9 1 1 2023-11-02T09:29:18.359 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test9' already exists 2023-11-02T09:29:18.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:29:19.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test9 size 2 2023-11-02T09:29:20.297 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 9 size to 2 2023-11-02T09:29:20.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:20.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test10 1 1 2023-11-02T09:29:20.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test10 1 1 2023-11-02T09:29:20.946 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test10' already exists 2023-11-02T09:29:20.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:29:21.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test10 size 2 2023-11-02T09:29:22.979 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 10 size to 2 2023-11-02T09:29:22.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:66: TEST_recovery_priority: sleep 5 2023-11-02T09:29:27.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:68: TEST_recovery_priority: wait_for_clean 2023-11-02T09:29:27.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T09:29:27.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T09:29:27.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T09:29:27.993 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-11-02T09:29:27.993 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T09:29:27.994 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T09:29:27.994 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T09:29:27.994 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T09:29:27.994 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T09:29:28.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-11-02T09:29:28.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T09:29:28.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T09:29:28.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T09:29:28.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T09:29:28.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T09:29:28.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T09:29:28.471 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T09:29:28.472 INFO:tasks.workunit.client.0.smithi116.stderr:2 2023-11-02T09:29:28.472 INFO:tasks.workunit.client.0.smithi116.stderr:3 2023-11-02T09:29:28.472 INFO:tasks.workunit.client.0.smithi116.stderr:4' 2023-11-02T09:29:28.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T09:29:28.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:29:28.473 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T09:29:28.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836498 2023-11-02T09:29:28.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836498 2023-11-02T09:29:28.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836498' 2023-11-02T09:29:28.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:29:28.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T09:29:28.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672976 2023-11-02T09:29:28.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672976 2023-11-02T09:29:28.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836498 1-42949672976' 2023-11-02T09:29:28.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:29:28.717 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T09:29:28.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509454 2023-11-02T09:29:28.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509454 2023-11-02T09:29:28.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454' 2023-11-02T09:29:28.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:29:28.832 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-11-02T09:29:28.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345931 2023-11-02T09:29:28.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345931 2023-11-02T09:29:28.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345931' 2023-11-02T09:29:28.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:29:28.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-11-02T09:29:29.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182409 2023-11-02T09:29:29.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182409 2023-11-02T09:29:29.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345931 4-107374182409' 2023-11-02T09:29:29.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:29:29.058 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836498 2023-11-02T09:29:29.058 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:29:29.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T09:29:29.059 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836498 2023-11-02T09:29:29.059 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:29:29.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836498 2023-11-02T09:29:29.060 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836498 2023-11-02T09:29:29.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2023-11-02T09:29:29.061 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:29:29.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836496 -lt 21474836498 2023-11-02T09:29:29.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T09:29:30.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T09:29:30.469 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:29:30.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836498 -lt 21474836498 2023-11-02T09:29:30.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:29:30.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672976 2023-11-02T09:29:30.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:29:30.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T09:29:30.896 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672976 2023-11-02T09:29:30.897 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:29:30.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672976 2023-11-02T09:29:30.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2023-11-02T09:29:30.898 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672976 2023-11-02T09:29:30.898 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T09:29:31.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672976 -lt 42949672976 2023-11-02T09:29:31.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:29:31.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509454 2023-11-02T09:29:31.313 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:29:31.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T09:29:31.314 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509454 2023-11-02T09:29:31.314 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:29:31.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509454 2023-11-02T09:29:31.315 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509454 2023-11-02T09:29:31.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2023-11-02T09:29:31.316 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T09:29:31.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509454 -lt 64424509454 2023-11-02T09:29:31.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:29:31.731 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345931 2023-11-02T09:29:31.731 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:29:31.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2023-11-02T09:29:31.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345931 2023-11-02T09:29:31.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:29:31.733 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 85899345931 2023-11-02T09:29:31.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345931 2023-11-02T09:29:31.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2023-11-02T09:29:31.734 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2023-11-02T09:29:32.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345932 -lt 85899345931 2023-11-02T09:29:32.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:29:32.148 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182409 2023-11-02T09:29:32.148 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:29:32.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2023-11-02T09:29:32.150 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182409 2023-11-02T09:29:32.150 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:29:32.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182409 2023-11-02T09:29:32.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2023-11-02T09:29:32.151 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.4 seq 107374182409 2023-11-02T09:29:32.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2023-11-02T09:29:32.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182410 -lt 107374182409 2023-11-02T09:29:32.558 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T09:29:32.558 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:29:32.558 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:29:33.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 10 == 0 2023-11-02T09:29:33.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:29:33.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:29:33.074 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:29:33.074 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:29:33.074 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:29:33.074 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:29:33.074 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:29:33.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=10 2023-11-02T09:29:33.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:29:33.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:29:33.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:29:33.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 10 = 10 2023-11-02T09:29:33.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T09:29:33.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T09:29:33.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:70: TEST_recovery_priority: ceph pg dump pgs 2023-11-02T09:29:34.324 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T09:29:34.325 INFO:tasks.workunit.client.0.smithi116.stdout:10.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-02T09:29:23.246955+0000 0'0 78:20 [2,1] 2 [2,1] 2 0'0 2023-11-02T09:29:20.645657+0000 0'0 2023-11-02T09:29:20.645657+0000 0 0 periodic scrub scheduled @ 2023-11-03T19:42:01.946843+0000 0 0 2023-11-02T09:29:34.325 INFO:tasks.workunit.client.0.smithi116.stdout:9.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-02T09:29:21.066054+0000 0'0 77:25 [1,2] 1 [1,2] 1 0'0 2023-11-02T09:29:18.024875+0000 0'0 2023-11-02T09:29:18.024875+0000 0 0 periodic scrub scheduled @ 2023-11-03T11:58:45.356767+0000 0 0 2023-11-02T09:29:34.325 INFO:tasks.workunit.client.0.smithi116.stdout:8.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-02T09:29:18.513284+0000 0'0 75:27 [3,4] 3 [3,4] 3 0'0 2023-11-02T09:29:15.083859+0000 0'0 2023-11-02T09:29:15.083859+0000 0 0 periodic scrub scheduled @ 2023-11-03T10:26:30.151248+0000 0 0 2023-11-02T09:29:34.325 INFO:tasks.workunit.client.0.smithi116.stdout:7.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-02T09:29:15.285843+0000 0'0 77:33 [1,4] 1 [1,4] 1 0'0 2023-11-02T09:29:12.218256+0000 0'0 2023-11-02T09:29:12.218256+0000 0 0 periodic scrub scheduled @ 2023-11-03T10:53:37.185676+0000 0 0 2023-11-02T09:29:34.326 INFO:tasks.workunit.client.0.smithi116.stdout:6.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-02T09:29:12.738925+0000 0'0 75:34 [4,2] 4 [4,2] 4 0'0 2023-11-02T09:29:09.742296+0000 0'0 2023-11-02T09:29:09.742296+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:23:07.880717+0000 0 0 2023-11-02T09:29:34.326 INFO:tasks.workunit.client.0.smithi116.stdout:5.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-02T09:29:09.579466+0000 0'0 75:39 [3,4] 3 [3,4] 3 0'0 2023-11-02T09:29:07.163956+0000 0'0 2023-11-02T09:29:07.163956+0000 0 0 periodic scrub scheduled @ 2023-11-03T15:39:30.223914+0000 0 0 2023-11-02T09:29:34.326 INFO:tasks.workunit.client.0.smithi116.stdout:4.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-02T09:29:07.911380+0000 0'0 75:44 [4,0] 4 [4,0] 4 0'0 2023-11-02T09:29:04.457030+0000 0'0 2023-11-02T09:29:04.457030+0000 0 0 periodic scrub scheduled @ 2023-11-03T14:30:00.552483+0000 0 0 2023-11-02T09:29:34.326 INFO:tasks.workunit.client.0.smithi116.stdout:3.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-02T09:29:04.851530+0000 0'0 77:43 [1,2] 1 [1,2] 1 0'0 2023-11-02T09:29:01.559968+0000 0'0 2023-11-02T09:29:01.559968+0000 0 0 periodic scrub scheduled @ 2023-11-03T15:31:58.079661+0000 0 0 2023-11-02T09:29:34.326 INFO:tasks.workunit.client.0.smithi116.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-02T09:29:01.787856+0000 0'0 75:50 [3,1] 3 [3,1] 3 0'0 2023-11-02T09:28:59.181089+0000 0'0 2023-11-02T09:28:59.181089+0000 0 0 periodic scrub scheduled @ 2023-11-03T17:26:22.507634+0000 0 0 2023-11-02T09:29:34.327 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-11-02T09:28:59.632794+0000 0'0 77:59 [1,0] 1 [1,0] 1 0'0 2023-11-02T09:28:56.349957+0000 0'0 2023-11-02T09:28:56.349957+0000 0 0 periodic scrub scheduled @ 2023-11-03T10:48:41.358143+0000 0 0 2023-11-02T09:29:34.327 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T09:29:34.327 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T09:29:34.327 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T09:29:34.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:74: TEST_recovery_priority: local PG1 2023-11-02T09:29:34.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:75: TEST_recovery_priority: local POOLNUM1 2023-11-02T09:29:34.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:76: TEST_recovery_priority: local pool1 2023-11-02T09:29:34.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:77: TEST_recovery_priority: local chk_osd1_1 2023-11-02T09:29:34.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:78: TEST_recovery_priority: local chk_osd1_2 2023-11-02T09:29:34.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:80: TEST_recovery_priority: local PG2 2023-11-02T09:29:34.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:81: TEST_recovery_priority: local POOLNUM2 2023-11-02T09:29:34.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:82: TEST_recovery_priority: local pool2 2023-11-02T09:29:34.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:83: TEST_recovery_priority: local chk_osd2 2023-11-02T09:29:34.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:85: TEST_recovery_priority: local PG3 2023-11-02T09:29:34.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: local POOLNUM3 2023-11-02T09:29:34.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:87: TEST_recovery_priority: local pool3 2023-11-02T09:29:34.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: seq 1 10 2023-11-02T09:29:34.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:34.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 1.0 --format=json 2023-11-02T09:29:34.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2023-11-02T09:29:34.704 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2023-11-02T09:29:34.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2023-11-02T09:29:34.705 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2023-11-02T09:29:34.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=0 2023-11-02T09:29:34.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z '' ']' 2023-11-02T09:29:34.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:96: TEST_recovery_priority: PG1=1.0 2023-11-02T09:29:34.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:97: TEST_recovery_priority: POOLNUM1=1 2023-11-02T09:29:34.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: pool1=test1 2023-11-02T09:29:34.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:99: TEST_recovery_priority: chk_osd1_1=1 2023-11-02T09:29:34.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:100: TEST_recovery_priority: chk_osd1_2=0 2023-11-02T09:29:34.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:34.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 2.0 --format=json 2023-11-02T09:29:34.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2023-11-02T09:29:35.114 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2023-11-02T09:29:35.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=3 2023-11-02T09:29:35.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2023-11-02T09:29:35.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=1 2023-11-02T09:29:35.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2023-11-02T09:29:35.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2023-11-02T09:29:35.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2023-11-02T09:29:35.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:35.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 3.0 --format=json 2023-11-02T09:29:35.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2023-11-02T09:29:35.527 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2023-11-02T09:29:35.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2023-11-02T09:29:35.527 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2023-11-02T09:29:35.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=2 2023-11-02T09:29:35.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2023-11-02T09:29:35.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2023-11-02T09:29:35.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:103: TEST_recovery_priority: PG2=3.0 2023-11-02T09:29:35.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: POOLNUM2=3 2023-11-02T09:29:35.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:105: TEST_recovery_priority: pool2=test3 2023-11-02T09:29:35.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:106: TEST_recovery_priority: chk_osd2=2 2023-11-02T09:29:35.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:35.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 4.0 --format=json 2023-11-02T09:29:35.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2023-11-02T09:29:35.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2023-11-02T09:29:35.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=4 2023-11-02T09:29:35.960 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2023-11-02T09:29:35.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=0 2023-11-02T09:29:35.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2023-11-02T09:29:35.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2023-11-02T09:29:35.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2023-11-02T09:29:35.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:35.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 5.0 --format=json 2023-11-02T09:29:35.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2023-11-02T09:29:36.328 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2023-11-02T09:29:36.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=3 2023-11-02T09:29:36.329 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2023-11-02T09:29:36.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=4 2023-11-02T09:29:36.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2023-11-02T09:29:36.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2023-11-02T09:29:36.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2023-11-02T09:29:36.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:36.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 6.0 --format=json 2023-11-02T09:29:36.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2023-11-02T09:29:36.698 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2023-11-02T09:29:36.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=4 2023-11-02T09:29:36.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2023-11-02T09:29:36.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=2 2023-11-02T09:29:36.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2023-11-02T09:29:36.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2023-11-02T09:29:36.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2023-11-02T09:29:36.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:36.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 7.0 --format=json 2023-11-02T09:29:36.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2023-11-02T09:29:37.072 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2023-11-02T09:29:37.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2023-11-02T09:29:37.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2023-11-02T09:29:37.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=4 2023-11-02T09:29:37.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2023-11-02T09:29:37.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2023-11-02T09:29:37.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2023-11-02T09:29:37.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:109: TEST_recovery_priority: PG3=7.0 2023-11-02T09:29:37.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:110: TEST_recovery_priority: POOLNUM3=7 2023-11-02T09:29:37.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:111: TEST_recovery_priority: pool3=test7 2023-11-02T09:29:37.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:112: TEST_recovery_priority: break 2023-11-02T09:29:37.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:115: TEST_recovery_priority: rm -f td/osd-recovery-prio/acting 2023-11-02T09:29:37.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:117: TEST_recovery_priority: '[' test3 = '' -o pool3 = '' ']' 2023-11-02T09:29:37.083 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: seq 1 10 2023-11-02T09:29:37.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:37.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2023-11-02T09:29:37.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:37.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2023-11-02T09:29:37.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test2 2023-11-02T09:29:37.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2023-11-02T09:29:37.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2023-11-02T09:29:37.606 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test2' does not exist 2023-11-02T09:29:37.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:37.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2023-11-02T09:29:37.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:37.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2023-11-02T09:29:37.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test4 2023-11-02T09:29:37.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test4 2023-11-02T09:29:37.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2023-11-02T09:29:38.141 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test4' does not exist 2023-11-02T09:29:38.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:38.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2023-11-02T09:29:38.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test5 2023-11-02T09:29:38.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test5 2023-11-02T09:29:38.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2023-11-02T09:29:38.741 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test5' does not exist 2023-11-02T09:29:38.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:38.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2023-11-02T09:29:38.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test6 2023-11-02T09:29:38.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test6 2023-11-02T09:29:38.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2023-11-02T09:29:39.304 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test6' does not exist 2023-11-02T09:29:39.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:39.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2023-11-02T09:29:39.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:39.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2023-11-02T09:29:39.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test8 2023-11-02T09:29:39.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test8 2023-11-02T09:29:39.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2023-11-02T09:29:39.992 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test8' does not exist 2023-11-02T09:29:40.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:40.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2023-11-02T09:29:40.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test9 2023-11-02T09:29:40.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test9 2023-11-02T09:29:40.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2023-11-02T09:29:40.760 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test9' does not exist 2023-11-02T09:29:40.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2023-11-02T09:29:40.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2023-11-02T09:29:40.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test10 2023-11-02T09:29:40.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test10 2023-11-02T09:29:40.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2023-11-02T09:29:41.330 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test10' does not exist 2023-11-02T09:29:41.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:131: TEST_recovery_priority: ceph osd pool set test3 size 1 --yes-i-really-mean-it 2023-11-02T09:29:42.159 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 3 size to 1 2023-11-02T09:29:42.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:132: TEST_recovery_priority: ceph osd pool set test7 size 1 --yes-i-really-mean-it 2023-11-02T09:29:43.249 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 7 size to 1 2023-11-02T09:29:43.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:133: TEST_recovery_priority: wait_for_clean 2023-11-02T09:29:43.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T09:29:43.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T09:29:43.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T09:29:43.261 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-11-02T09:29:43.262 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T09:29:43.262 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T09:29:43.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T09:29:43.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T09:29:43.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T09:29:43.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-11-02T09:29:43.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T09:29:43.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T09:29:43.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T09:29:43.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T09:29:43.344 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T09:29:43.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T09:29:43.813 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T09:29:43.813 INFO:tasks.workunit.client.0.smithi116.stderr:2 2023-11-02T09:29:43.814 INFO:tasks.workunit.client.0.smithi116.stderr:3 2023-11-02T09:29:43.814 INFO:tasks.workunit.client.0.smithi116.stderr:4' 2023-11-02T09:29:43.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T09:29:43.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:29:43.814 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T09:29:44.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836503 2023-11-02T09:29:44.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836503 2023-11-02T09:29:44.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836503' 2023-11-02T09:29:44.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:29:44.003 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T09:29:44.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672981 2023-11-02T09:29:44.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672981 2023-11-02T09:29:44.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836503 1-42949672981' 2023-11-02T09:29:44.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:29:44.116 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T09:29:44.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509459 2023-11-02T09:29:44.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509459 2023-11-02T09:29:44.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836503 1-42949672981 2-64424509459' 2023-11-02T09:29:44.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:29:44.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-11-02T09:29:44.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345937 2023-11-02T09:29:44.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345937 2023-11-02T09:29:44.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836503 1-42949672981 2-64424509459 3-85899345937' 2023-11-02T09:29:44.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:29:44.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-11-02T09:29:44.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182414 2023-11-02T09:29:44.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182414 2023-11-02T09:29:44.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836503 1-42949672981 2-64424509459 3-85899345937 4-107374182414' 2023-11-02T09:29:44.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:29:44.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836503 2023-11-02T09:29:44.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:29:44.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T09:29:44.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836503 2023-11-02T09:29:44.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:29:44.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836503 2023-11-02T09:29:44.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2023-11-02T09:29:44.459 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836503 2023-11-02T09:29:44.460 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:29:44.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836503 -lt 21474836503 2023-11-02T09:29:44.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:29:44.876 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672981 2023-11-02T09:29:44.877 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:29:44.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T09:29:44.878 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672981 2023-11-02T09:29:44.878 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:29:44.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672981 2023-11-02T09:29:44.879 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672981 2023-11-02T09:29:44.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2023-11-02T09:29:44.880 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T09:29:45.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672981 -lt 42949672981 2023-11-02T09:29:45.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:29:45.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509459 2023-11-02T09:29:45.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:29:45.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T09:29:45.296 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509459 2023-11-02T09:29:45.296 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:29:45.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509459 2023-11-02T09:29:45.297 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509459 2023-11-02T09:29:45.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2023-11-02T09:29:45.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T09:29:45.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509459 -lt 64424509459 2023-11-02T09:29:45.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:29:45.671 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345937 2023-11-02T09:29:45.672 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:29:45.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2023-11-02T09:29:45.673 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345937 2023-11-02T09:29:45.673 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:29:45.673 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 85899345937 2023-11-02T09:29:45.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345937 2023-11-02T09:29:45.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345937' 2023-11-02T09:29:45.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2023-11-02T09:29:46.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345934 -lt 85899345937 2023-11-02T09:29:46.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T09:29:47.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T09:29:47.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2023-11-02T09:29:47.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345937 -lt 85899345937 2023-11-02T09:29:47.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:29:47.466 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182414 2023-11-02T09:29:47.466 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:29:47.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2023-11-02T09:29:47.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182414 2023-11-02T09:29:47.468 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:29:47.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182414 2023-11-02T09:29:47.468 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.4 seq 107374182414 2023-11-02T09:29:47.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2023-11-02T09:29:47.469 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2023-11-02T09:29:47.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182415 -lt 107374182414 2023-11-02T09:29:47.838 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T09:29:47.838 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:29:47.839 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:29:48.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2023-11-02T09:29:48.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:29:48.337 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:29:48.338 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:29:48.338 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:29:48.338 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:29:48.338 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:29:48.338 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:29:48.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2023-11-02T09:29:48.767 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:29:48.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:29:48.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:29:49.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2023-11-02T09:29:49.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T09:29:49.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T09:29:49.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:135: TEST_recovery_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2023-11-02T09:29:49.297 INFO:tasks.workunit.client.0.smithi116.stderr:10+0 records in 2023-11-02T09:29:49.298 INFO:tasks.workunit.client.0.smithi116.stderr:10+0 records out 2023-11-02T09:29:49.298 INFO:tasks.workunit.client.0.smithi116.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0303642 s, 345 MB/s 2023-11-02T09:29:49.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: p=1 2023-11-02T09:29:49.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2023-11-02T09:29:49.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2023-11-02T09:29:49.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:29:49.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2023-11-02T09:29:49.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:29:49.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2023-11-02T09:29:50.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:29:50.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2023-11-02T09:29:51.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:29:51.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2023-11-02T09:29:51.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:29:51.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2023-11-02T09:29:52.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:29:52.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2023-11-02T09:29:53.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:29:53.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2023-11-02T09:29:53.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:29:53.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2023-11-02T09:29:54.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:29:54.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2023-11-02T09:29:55.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:29:55.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2023-11-02T09:29:55.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:29:55.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2023-11-02T09:29:56.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:29:56.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2023-11-02T09:29:57.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:29:57.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2023-11-02T09:29:57.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:29:57.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2023-11-02T09:29:58.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:29:58.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2023-11-02T09:29:59.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:29:59.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2023-11-02T09:29:59.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:29:59.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2023-11-02T09:30:00.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:00.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2023-11-02T09:30:01.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:01.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2023-11-02T09:30:01.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:01.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2023-11-02T09:30:02.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:02.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2023-11-02T09:30:02.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:02.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2023-11-02T09:30:03.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:03.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2023-11-02T09:30:04.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:04.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2023-11-02T09:30:04.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:04.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2023-11-02T09:30:05.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:05.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2023-11-02T09:30:06.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:06.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2023-11-02T09:30:06.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:06.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2023-11-02T09:30:07.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:07.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2023-11-02T09:30:07.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:07.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2023-11-02T09:30:08.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:08.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2023-11-02T09:30:09.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:09.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2023-11-02T09:30:09.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:09.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2023-11-02T09:30:10.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:10.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2023-11-02T09:30:11.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:11.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2023-11-02T09:30:11.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:11.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2023-11-02T09:30:12.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:12.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2023-11-02T09:30:13.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:13.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2023-11-02T09:30:13.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:13.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2023-11-02T09:30:14.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:14.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2023-11-02T09:30:14.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:14.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2023-11-02T09:30:15.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:15.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2023-11-02T09:30:16.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:16.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2023-11-02T09:30:16.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:16.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2023-11-02T09:30:17.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:17.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2023-11-02T09:30:18.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:18.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2023-11-02T09:30:18.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:18.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2023-11-02T09:30:19.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:19.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2023-11-02T09:30:20.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:20.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2023-11-02T09:30:20.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:20.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2023-11-02T09:30:21.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:21.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2023-11-02T09:30:21.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:21.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2023-11-02T09:30:22.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:22.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2023-11-02T09:30:23.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:23.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2023-11-02T09:30:24.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:24.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2023-11-02T09:30:24.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:24.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2023-11-02T09:30:25.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:25.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2023-11-02T09:30:26.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:26.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2023-11-02T09:30:26.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:26.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2023-11-02T09:30:27.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:27.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2023-11-02T09:30:28.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:28.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2023-11-02T09:30:28.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:28.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2023-11-02T09:30:29.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:29.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2023-11-02T09:30:30.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:30.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2023-11-02T09:30:30.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:30.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2023-11-02T09:30:31.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:31.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2023-11-02T09:30:31.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:31.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2023-11-02T09:30:32.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:32.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2023-11-02T09:30:33.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:33.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2023-11-02T09:30:33.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:33.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2023-11-02T09:30:34.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:34.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2023-11-02T09:30:35.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:35.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2023-11-02T09:30:35.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:35.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2023-11-02T09:30:36.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:36.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2023-11-02T09:30:37.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:37.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2023-11-02T09:30:37.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:37.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2023-11-02T09:30:38.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:38.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2023-11-02T09:30:39.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:39.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2023-11-02T09:30:39.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:39.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2023-11-02T09:30:40.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:40.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2023-11-02T09:30:41.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:41.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2023-11-02T09:30:41.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:41.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2023-11-02T09:30:42.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:42.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2023-11-02T09:30:43.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:43.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2023-11-02T09:30:43.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:43.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2023-11-02T09:30:44.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:44.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2023-11-02T09:30:44.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:44.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2023-11-02T09:30:45.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:45.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2023-11-02T09:30:46.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:46.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2023-11-02T09:30:46.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:46.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2023-11-02T09:30:47.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:47.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2023-11-02T09:30:48.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:48.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2023-11-02T09:30:49.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:49.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2023-11-02T09:30:49.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:49.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2023-11-02T09:30:50.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:50.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2023-11-02T09:30:50.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:50.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2023-11-02T09:30:51.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:51.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2023-11-02T09:30:52.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:52.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2023-11-02T09:30:52.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:52.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2023-11-02T09:30:53.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:53.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2023-11-02T09:30:54.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:54.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2023-11-02T09:30:54.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2023-11-02T09:30:55.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:55.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2023-11-02T09:30:56.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:56.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2023-11-02T09:30:56.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:56.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2023-11-02T09:30:57.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:57.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2023-11-02T09:30:57.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:57.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2023-11-02T09:30:58.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:58.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2023-11-02T09:30:59.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:59.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2023-11-02T09:30:59.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:30:59.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2023-11-02T09:31:00.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:00.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2023-11-02T09:31:01.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:01.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2023-11-02T09:31:01.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:01.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2023-11-02T09:31:02.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:02.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2023-11-02T09:31:03.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:03.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2023-11-02T09:31:03.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:03.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2023-11-02T09:31:04.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:04.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2023-11-02T09:31:05.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:05.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2023-11-02T09:31:05.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:05.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2023-11-02T09:31:06.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:06.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2023-11-02T09:31:07.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:07.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2023-11-02T09:31:07.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:07.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2023-11-02T09:31:08.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:08.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2023-11-02T09:31:09.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:09.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2023-11-02T09:31:09.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:09.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2023-11-02T09:31:10.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:10.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2023-11-02T09:31:10.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:10.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2023-11-02T09:31:11.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:11.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2023-11-02T09:31:12.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:12.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2023-11-02T09:31:12.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:12.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2023-11-02T09:31:13.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:13.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2023-11-02T09:31:14.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:14.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2023-11-02T09:31:14.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:14.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2023-11-02T09:31:15.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:15.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2023-11-02T09:31:16.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:16.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2023-11-02T09:31:16.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:16.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2023-11-02T09:31:17.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:17.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2023-11-02T09:31:18.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:18.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2023-11-02T09:31:18.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:18.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2023-11-02T09:31:19.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:19.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2023-11-02T09:31:20.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:20.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2023-11-02T09:31:20.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:20.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2023-11-02T09:31:21.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:21.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2023-11-02T09:31:21.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:21.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2023-11-02T09:31:22.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:22.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2023-11-02T09:31:23.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:23.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2023-11-02T09:31:23.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:23.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2023-11-02T09:31:24.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:24.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2023-11-02T09:31:25.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:25.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2023-11-02T09:31:26.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:26.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2023-11-02T09:31:26.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:26.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2023-11-02T09:31:27.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:27.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2023-11-02T09:31:28.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:28.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2023-11-02T09:31:28.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:28.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2023-11-02T09:31:29.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:29.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2023-11-02T09:31:29.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:29.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2023-11-02T09:31:30.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:30.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2023-11-02T09:31:31.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:31.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2023-11-02T09:31:32.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:32.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2023-11-02T09:31:32.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:32.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2023-11-02T09:31:33.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:33.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2023-11-02T09:31:33.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:33.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2023-11-02T09:31:34.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:34.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2023-11-02T09:31:35.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:35.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2023-11-02T09:31:35.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:35.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2023-11-02T09:31:36.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:36.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2023-11-02T09:31:37.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:37.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2023-11-02T09:31:37.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:37.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2023-11-02T09:31:38.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:38.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2023-11-02T09:31:39.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:39.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2023-11-02T09:31:39.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:39.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2023-11-02T09:31:40.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:40.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2023-11-02T09:31:41.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:41.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2023-11-02T09:31:41.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:41.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2023-11-02T09:31:42.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:42.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2023-11-02T09:31:43.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:43.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2023-11-02T09:31:43.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:43.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2023-11-02T09:31:44.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:44.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2023-11-02T09:31:45.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:45.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2023-11-02T09:31:45.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:45.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2023-11-02T09:31:46.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:46.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2023-11-02T09:31:47.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:47.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2023-11-02T09:31:47.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:47.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2023-11-02T09:31:48.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:48.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2023-11-02T09:31:49.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:49.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2023-11-02T09:31:49.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:49.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2023-11-02T09:31:50.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:50.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2023-11-02T09:31:51.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:51.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2023-11-02T09:31:51.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:51.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2023-11-02T09:31:52.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:52.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2023-11-02T09:31:53.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:53.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2023-11-02T09:31:53.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:53.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2023-11-02T09:31:54.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:54.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2023-11-02T09:31:55.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:55.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2023-11-02T09:31:55.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:55.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2023-11-02T09:31:56.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:56.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2023-11-02T09:31:57.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:57.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2023-11-02T09:31:57.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:57.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2023-11-02T09:31:58.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:58.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2023-11-02T09:31:59.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:59.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2023-11-02T09:31:59.867 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 1 + 1 2023-11-02T09:31:59.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=2 2023-11-02T09:31:59.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2023-11-02T09:31:59.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2023-11-02T09:31:59.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:31:59.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj1-p2 td/osd-recovery-prio/data 2023-11-02T09:32:00.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:00.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj2-p2 td/osd-recovery-prio/data 2023-11-02T09:32:00.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:00.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj3-p2 td/osd-recovery-prio/data 2023-11-02T09:32:01.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:01.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj4-p2 td/osd-recovery-prio/data 2023-11-02T09:32:01.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:01.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj5-p2 td/osd-recovery-prio/data 2023-11-02T09:32:01.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:01.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj6-p2 td/osd-recovery-prio/data 2023-11-02T09:32:02.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:02.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj7-p2 td/osd-recovery-prio/data 2023-11-02T09:32:02.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:02.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj8-p2 td/osd-recovery-prio/data 2023-11-02T09:32:02.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:02.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj9-p2 td/osd-recovery-prio/data 2023-11-02T09:32:03.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:03.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj10-p2 td/osd-recovery-prio/data 2023-11-02T09:32:03.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:03.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj11-p2 td/osd-recovery-prio/data 2023-11-02T09:32:03.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:03.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj12-p2 td/osd-recovery-prio/data 2023-11-02T09:32:04.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:04.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj13-p2 td/osd-recovery-prio/data 2023-11-02T09:32:04.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:04.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj14-p2 td/osd-recovery-prio/data 2023-11-02T09:32:04.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:04.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj15-p2 td/osd-recovery-prio/data 2023-11-02T09:32:05.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:05.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj16-p2 td/osd-recovery-prio/data 2023-11-02T09:32:05.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:05.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj17-p2 td/osd-recovery-prio/data 2023-11-02T09:32:05.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:05.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj18-p2 td/osd-recovery-prio/data 2023-11-02T09:32:06.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:06.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj19-p2 td/osd-recovery-prio/data 2023-11-02T09:32:06.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:06.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj20-p2 td/osd-recovery-prio/data 2023-11-02T09:32:07.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:07.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj21-p2 td/osd-recovery-prio/data 2023-11-02T09:32:07.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:07.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj22-p2 td/osd-recovery-prio/data 2023-11-02T09:32:07.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:07.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj23-p2 td/osd-recovery-prio/data 2023-11-02T09:32:08.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:08.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj24-p2 td/osd-recovery-prio/data 2023-11-02T09:32:08.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:08.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj25-p2 td/osd-recovery-prio/data 2023-11-02T09:32:08.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:08.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj26-p2 td/osd-recovery-prio/data 2023-11-02T09:32:09.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:09.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj27-p2 td/osd-recovery-prio/data 2023-11-02T09:32:09.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:09.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj28-p2 td/osd-recovery-prio/data 2023-11-02T09:32:09.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:09.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj29-p2 td/osd-recovery-prio/data 2023-11-02T09:32:10.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:10.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj30-p2 td/osd-recovery-prio/data 2023-11-02T09:32:10.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:10.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj31-p2 td/osd-recovery-prio/data 2023-11-02T09:32:11.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:11.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj32-p2 td/osd-recovery-prio/data 2023-11-02T09:32:11.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:11.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj33-p2 td/osd-recovery-prio/data 2023-11-02T09:32:11.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:11.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj34-p2 td/osd-recovery-prio/data 2023-11-02T09:32:12.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:12.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj35-p2 td/osd-recovery-prio/data 2023-11-02T09:32:12.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:12.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj36-p2 td/osd-recovery-prio/data 2023-11-02T09:32:12.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:12.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj37-p2 td/osd-recovery-prio/data 2023-11-02T09:32:13.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:13.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj38-p2 td/osd-recovery-prio/data 2023-11-02T09:32:13.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:13.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj39-p2 td/osd-recovery-prio/data 2023-11-02T09:32:13.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:13.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj40-p2 td/osd-recovery-prio/data 2023-11-02T09:32:14.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:14.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj41-p2 td/osd-recovery-prio/data 2023-11-02T09:32:14.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:14.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj42-p2 td/osd-recovery-prio/data 2023-11-02T09:32:14.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:14.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj43-p2 td/osd-recovery-prio/data 2023-11-02T09:32:15.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:15.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj44-p2 td/osd-recovery-prio/data 2023-11-02T09:32:15.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:15.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj45-p2 td/osd-recovery-prio/data 2023-11-02T09:32:16.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:16.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj46-p2 td/osd-recovery-prio/data 2023-11-02T09:32:16.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:16.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj47-p2 td/osd-recovery-prio/data 2023-11-02T09:32:16.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:16.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj48-p2 td/osd-recovery-prio/data 2023-11-02T09:32:17.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:17.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj49-p2 td/osd-recovery-prio/data 2023-11-02T09:32:17.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:17.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj50-p2 td/osd-recovery-prio/data 2023-11-02T09:32:17.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:17.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj51-p2 td/osd-recovery-prio/data 2023-11-02T09:32:18.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:18.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj52-p2 td/osd-recovery-prio/data 2023-11-02T09:32:18.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:18.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj53-p2 td/osd-recovery-prio/data 2023-11-02T09:32:18.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:18.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj54-p2 td/osd-recovery-prio/data 2023-11-02T09:32:19.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:19.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj55-p2 td/osd-recovery-prio/data 2023-11-02T09:32:19.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:19.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj56-p2 td/osd-recovery-prio/data 2023-11-02T09:32:19.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:19.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj57-p2 td/osd-recovery-prio/data 2023-11-02T09:32:20.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:20.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj58-p2 td/osd-recovery-prio/data 2023-11-02T09:32:20.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:20.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj59-p2 td/osd-recovery-prio/data 2023-11-02T09:32:21.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:21.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj60-p2 td/osd-recovery-prio/data 2023-11-02T09:32:21.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:21.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj61-p2 td/osd-recovery-prio/data 2023-11-02T09:32:21.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:21.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj62-p2 td/osd-recovery-prio/data 2023-11-02T09:32:22.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:22.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj63-p2 td/osd-recovery-prio/data 2023-11-02T09:32:22.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:22.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj64-p2 td/osd-recovery-prio/data 2023-11-02T09:32:22.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:22.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj65-p2 td/osd-recovery-prio/data 2023-11-02T09:32:23.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:23.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj66-p2 td/osd-recovery-prio/data 2023-11-02T09:32:23.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:23.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj67-p2 td/osd-recovery-prio/data 2023-11-02T09:32:23.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:23.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj68-p2 td/osd-recovery-prio/data 2023-11-02T09:32:24.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:24.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj69-p2 td/osd-recovery-prio/data 2023-11-02T09:32:24.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:24.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj70-p2 td/osd-recovery-prio/data 2023-11-02T09:32:25.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:25.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj71-p2 td/osd-recovery-prio/data 2023-11-02T09:32:25.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:25.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj72-p2 td/osd-recovery-prio/data 2023-11-02T09:32:25.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:25.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj73-p2 td/osd-recovery-prio/data 2023-11-02T09:32:26.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:26.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj74-p2 td/osd-recovery-prio/data 2023-11-02T09:32:26.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:26.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj75-p2 td/osd-recovery-prio/data 2023-11-02T09:32:27.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:27.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj76-p2 td/osd-recovery-prio/data 2023-11-02T09:32:27.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:27.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj77-p2 td/osd-recovery-prio/data 2023-11-02T09:32:27.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:27.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj78-p2 td/osd-recovery-prio/data 2023-11-02T09:32:28.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:28.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj79-p2 td/osd-recovery-prio/data 2023-11-02T09:32:28.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:28.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj80-p2 td/osd-recovery-prio/data 2023-11-02T09:32:29.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:29.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj81-p2 td/osd-recovery-prio/data 2023-11-02T09:32:29.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:29.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj82-p2 td/osd-recovery-prio/data 2023-11-02T09:32:29.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:29.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj83-p2 td/osd-recovery-prio/data 2023-11-02T09:32:30.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:30.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj84-p2 td/osd-recovery-prio/data 2023-11-02T09:32:30.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:30.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj85-p2 td/osd-recovery-prio/data 2023-11-02T09:32:30.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:30.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj86-p2 td/osd-recovery-prio/data 2023-11-02T09:32:31.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:31.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj87-p2 td/osd-recovery-prio/data 2023-11-02T09:32:31.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:31.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj88-p2 td/osd-recovery-prio/data 2023-11-02T09:32:31.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:31.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj89-p2 td/osd-recovery-prio/data 2023-11-02T09:32:32.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:32.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj90-p2 td/osd-recovery-prio/data 2023-11-02T09:32:32.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:32.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj91-p2 td/osd-recovery-prio/data 2023-11-02T09:32:33.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:33.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj92-p2 td/osd-recovery-prio/data 2023-11-02T09:32:33.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:33.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj93-p2 td/osd-recovery-prio/data 2023-11-02T09:32:33.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:33.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj94-p2 td/osd-recovery-prio/data 2023-11-02T09:32:34.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:34.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj95-p2 td/osd-recovery-prio/data 2023-11-02T09:32:34.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:34.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj96-p2 td/osd-recovery-prio/data 2023-11-02T09:32:34.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:34.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj97-p2 td/osd-recovery-prio/data 2023-11-02T09:32:35.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:35.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj98-p2 td/osd-recovery-prio/data 2023-11-02T09:32:35.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:35.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj99-p2 td/osd-recovery-prio/data 2023-11-02T09:32:35.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:35.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj100-p2 td/osd-recovery-prio/data 2023-11-02T09:32:36.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:36.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj101-p2 td/osd-recovery-prio/data 2023-11-02T09:32:36.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:36.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj102-p2 td/osd-recovery-prio/data 2023-11-02T09:32:37.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:37.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj103-p2 td/osd-recovery-prio/data 2023-11-02T09:32:37.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:37.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj104-p2 td/osd-recovery-prio/data 2023-11-02T09:32:37.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:37.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj105-p2 td/osd-recovery-prio/data 2023-11-02T09:32:38.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:38.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj106-p2 td/osd-recovery-prio/data 2023-11-02T09:32:38.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:38.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj107-p2 td/osd-recovery-prio/data 2023-11-02T09:32:38.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:38.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj108-p2 td/osd-recovery-prio/data 2023-11-02T09:32:39.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:39.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj109-p2 td/osd-recovery-prio/data 2023-11-02T09:32:39.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:39.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj110-p2 td/osd-recovery-prio/data 2023-11-02T09:32:39.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:39.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj111-p2 td/osd-recovery-prio/data 2023-11-02T09:32:40.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:40.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj112-p2 td/osd-recovery-prio/data 2023-11-02T09:32:40.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:40.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj113-p2 td/osd-recovery-prio/data 2023-11-02T09:32:41.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:41.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj114-p2 td/osd-recovery-prio/data 2023-11-02T09:32:41.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:41.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj115-p2 td/osd-recovery-prio/data 2023-11-02T09:32:41.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:41.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj116-p2 td/osd-recovery-prio/data 2023-11-02T09:32:42.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:42.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj117-p2 td/osd-recovery-prio/data 2023-11-02T09:32:42.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:42.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj118-p2 td/osd-recovery-prio/data 2023-11-02T09:32:42.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:42.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj119-p2 td/osd-recovery-prio/data 2023-11-02T09:32:43.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:43.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj120-p2 td/osd-recovery-prio/data 2023-11-02T09:32:43.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:43.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj121-p2 td/osd-recovery-prio/data 2023-11-02T09:32:43.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:43.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj122-p2 td/osd-recovery-prio/data 2023-11-02T09:32:44.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:44.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj123-p2 td/osd-recovery-prio/data 2023-11-02T09:32:44.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:44.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj124-p2 td/osd-recovery-prio/data 2023-11-02T09:32:45.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:45.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj125-p2 td/osd-recovery-prio/data 2023-11-02T09:32:45.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:45.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj126-p2 td/osd-recovery-prio/data 2023-11-02T09:32:45.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:45.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj127-p2 td/osd-recovery-prio/data 2023-11-02T09:32:46.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:46.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj128-p2 td/osd-recovery-prio/data 2023-11-02T09:32:46.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:46.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj129-p2 td/osd-recovery-prio/data 2023-11-02T09:32:47.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:47.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj130-p2 td/osd-recovery-prio/data 2023-11-02T09:32:47.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:47.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj131-p2 td/osd-recovery-prio/data 2023-11-02T09:32:47.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:47.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj132-p2 td/osd-recovery-prio/data 2023-11-02T09:32:48.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:48.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj133-p2 td/osd-recovery-prio/data 2023-11-02T09:32:48.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:48.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj134-p2 td/osd-recovery-prio/data 2023-11-02T09:32:48.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:48.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj135-p2 td/osd-recovery-prio/data 2023-11-02T09:32:49.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:49.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj136-p2 td/osd-recovery-prio/data 2023-11-02T09:32:49.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:49.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj137-p2 td/osd-recovery-prio/data 2023-11-02T09:32:49.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:49.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj138-p2 td/osd-recovery-prio/data 2023-11-02T09:32:50.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:50.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj139-p2 td/osd-recovery-prio/data 2023-11-02T09:32:50.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:50.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj140-p2 td/osd-recovery-prio/data 2023-11-02T09:32:51.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:51.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj141-p2 td/osd-recovery-prio/data 2023-11-02T09:32:51.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:51.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj142-p2 td/osd-recovery-prio/data 2023-11-02T09:32:51.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:51.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj143-p2 td/osd-recovery-prio/data 2023-11-02T09:32:52.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:52.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj144-p2 td/osd-recovery-prio/data 2023-11-02T09:32:52.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:52.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj145-p2 td/osd-recovery-prio/data 2023-11-02T09:32:52.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:52.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj146-p2 td/osd-recovery-prio/data 2023-11-02T09:32:53.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:53.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj147-p2 td/osd-recovery-prio/data 2023-11-02T09:32:53.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:53.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj148-p2 td/osd-recovery-prio/data 2023-11-02T09:32:53.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:53.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj149-p2 td/osd-recovery-prio/data 2023-11-02T09:32:54.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:54.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj150-p2 td/osd-recovery-prio/data 2023-11-02T09:32:54.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:54.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj151-p2 td/osd-recovery-prio/data 2023-11-02T09:32:54.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:54.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj152-p2 td/osd-recovery-prio/data 2023-11-02T09:32:55.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:55.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj153-p2 td/osd-recovery-prio/data 2023-11-02T09:32:55.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:55.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj154-p2 td/osd-recovery-prio/data 2023-11-02T09:32:55.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:55.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj155-p2 td/osd-recovery-prio/data 2023-11-02T09:32:56.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:56.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj156-p2 td/osd-recovery-prio/data 2023-11-02T09:32:56.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:56.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj157-p2 td/osd-recovery-prio/data 2023-11-02T09:32:57.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:57.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj158-p2 td/osd-recovery-prio/data 2023-11-02T09:32:57.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:57.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj159-p2 td/osd-recovery-prio/data 2023-11-02T09:32:57.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:57.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj160-p2 td/osd-recovery-prio/data 2023-11-02T09:32:58.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:58.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj161-p2 td/osd-recovery-prio/data 2023-11-02T09:32:58.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:58.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj162-p2 td/osd-recovery-prio/data 2023-11-02T09:32:59.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:59.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj163-p2 td/osd-recovery-prio/data 2023-11-02T09:32:59.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:59.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj164-p2 td/osd-recovery-prio/data 2023-11-02T09:32:59.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:32:59.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj165-p2 td/osd-recovery-prio/data 2023-11-02T09:33:00.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:00.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj166-p2 td/osd-recovery-prio/data 2023-11-02T09:33:00.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:00.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj167-p2 td/osd-recovery-prio/data 2023-11-02T09:33:00.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:00.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj168-p2 td/osd-recovery-prio/data 2023-11-02T09:33:01.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:01.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj169-p2 td/osd-recovery-prio/data 2023-11-02T09:33:01.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:01.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj170-p2 td/osd-recovery-prio/data 2023-11-02T09:33:01.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:01.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj171-p2 td/osd-recovery-prio/data 2023-11-02T09:33:02.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:02.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj172-p2 td/osd-recovery-prio/data 2023-11-02T09:33:02.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:02.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj173-p2 td/osd-recovery-prio/data 2023-11-02T09:33:03.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:03.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj174-p2 td/osd-recovery-prio/data 2023-11-02T09:33:03.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:03.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj175-p2 td/osd-recovery-prio/data 2023-11-02T09:33:03.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:03.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj176-p2 td/osd-recovery-prio/data 2023-11-02T09:33:04.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:04.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj177-p2 td/osd-recovery-prio/data 2023-11-02T09:33:04.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:04.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj178-p2 td/osd-recovery-prio/data 2023-11-02T09:33:04.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:04.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj179-p2 td/osd-recovery-prio/data 2023-11-02T09:33:05.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:05.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj180-p2 td/osd-recovery-prio/data 2023-11-02T09:33:05.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:05.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj181-p2 td/osd-recovery-prio/data 2023-11-02T09:33:05.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:05.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj182-p2 td/osd-recovery-prio/data 2023-11-02T09:33:06.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:06.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj183-p2 td/osd-recovery-prio/data 2023-11-02T09:33:06.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:06.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj184-p2 td/osd-recovery-prio/data 2023-11-02T09:33:06.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:06.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj185-p2 td/osd-recovery-prio/data 2023-11-02T09:33:07.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:07.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj186-p2 td/osd-recovery-prio/data 2023-11-02T09:33:07.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:07.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj187-p2 td/osd-recovery-prio/data 2023-11-02T09:33:07.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:07.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj188-p2 td/osd-recovery-prio/data 2023-11-02T09:33:08.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:08.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj189-p2 td/osd-recovery-prio/data 2023-11-02T09:33:08.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:08.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj190-p2 td/osd-recovery-prio/data 2023-11-02T09:33:09.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:09.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj191-p2 td/osd-recovery-prio/data 2023-11-02T09:33:09.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:09.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj192-p2 td/osd-recovery-prio/data 2023-11-02T09:33:09.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:09.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj193-p2 td/osd-recovery-prio/data 2023-11-02T09:33:10.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:10.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj194-p2 td/osd-recovery-prio/data 2023-11-02T09:33:10.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:10.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj195-p2 td/osd-recovery-prio/data 2023-11-02T09:33:10.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:10.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj196-p2 td/osd-recovery-prio/data 2023-11-02T09:33:11.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:11.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj197-p2 td/osd-recovery-prio/data 2023-11-02T09:33:11.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:11.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj198-p2 td/osd-recovery-prio/data 2023-11-02T09:33:11.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:11.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj199-p2 td/osd-recovery-prio/data 2023-11-02T09:33:12.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:12.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj200-p2 td/osd-recovery-prio/data 2023-11-02T09:33:12.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 2 + 1 2023-11-02T09:33:12.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=3 2023-11-02T09:33:12.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2023-11-02T09:33:12.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2023-11-02T09:33:12.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:12.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj1-p3 td/osd-recovery-prio/data 2023-11-02T09:33:12.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:12.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj2-p3 td/osd-recovery-prio/data 2023-11-02T09:33:13.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:13.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj3-p3 td/osd-recovery-prio/data 2023-11-02T09:33:13.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:13.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj4-p3 td/osd-recovery-prio/data 2023-11-02T09:33:13.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:13.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj5-p3 td/osd-recovery-prio/data 2023-11-02T09:33:14.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:14.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj6-p3 td/osd-recovery-prio/data 2023-11-02T09:33:14.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:14.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj7-p3 td/osd-recovery-prio/data 2023-11-02T09:33:15.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:15.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj8-p3 td/osd-recovery-prio/data 2023-11-02T09:33:15.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:15.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj9-p3 td/osd-recovery-prio/data 2023-11-02T09:33:15.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:15.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj10-p3 td/osd-recovery-prio/data 2023-11-02T09:33:16.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:16.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj11-p3 td/osd-recovery-prio/data 2023-11-02T09:33:16.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:16.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj12-p3 td/osd-recovery-prio/data 2023-11-02T09:33:16.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:16.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj13-p3 td/osd-recovery-prio/data 2023-11-02T09:33:17.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:17.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj14-p3 td/osd-recovery-prio/data 2023-11-02T09:33:17.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:17.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj15-p3 td/osd-recovery-prio/data 2023-11-02T09:33:17.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:17.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj16-p3 td/osd-recovery-prio/data 2023-11-02T09:33:18.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:18.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj17-p3 td/osd-recovery-prio/data 2023-11-02T09:33:18.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:18.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj18-p3 td/osd-recovery-prio/data 2023-11-02T09:33:19.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:19.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj19-p3 td/osd-recovery-prio/data 2023-11-02T09:33:19.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:19.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj20-p3 td/osd-recovery-prio/data 2023-11-02T09:33:19.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:19.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj21-p3 td/osd-recovery-prio/data 2023-11-02T09:33:20.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:20.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj22-p3 td/osd-recovery-prio/data 2023-11-02T09:33:20.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:20.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj23-p3 td/osd-recovery-prio/data 2023-11-02T09:33:20.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:20.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj24-p3 td/osd-recovery-prio/data 2023-11-02T09:33:21.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:21.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj25-p3 td/osd-recovery-prio/data 2023-11-02T09:33:21.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:21.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj26-p3 td/osd-recovery-prio/data 2023-11-02T09:33:21.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:21.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj27-p3 td/osd-recovery-prio/data 2023-11-02T09:33:22.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:22.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj28-p3 td/osd-recovery-prio/data 2023-11-02T09:33:22.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:22.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj29-p3 td/osd-recovery-prio/data 2023-11-02T09:33:22.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:22.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj30-p3 td/osd-recovery-prio/data 2023-11-02T09:33:23.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:23.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj31-p3 td/osd-recovery-prio/data 2023-11-02T09:33:23.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:23.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj32-p3 td/osd-recovery-prio/data 2023-11-02T09:33:23.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:23.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj33-p3 td/osd-recovery-prio/data 2023-11-02T09:33:24.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:24.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj34-p3 td/osd-recovery-prio/data 2023-11-02T09:33:24.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:24.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj35-p3 td/osd-recovery-prio/data 2023-11-02T09:33:25.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:25.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj36-p3 td/osd-recovery-prio/data 2023-11-02T09:33:25.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:25.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj37-p3 td/osd-recovery-prio/data 2023-11-02T09:33:25.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:25.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj38-p3 td/osd-recovery-prio/data 2023-11-02T09:33:26.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:26.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj39-p3 td/osd-recovery-prio/data 2023-11-02T09:33:26.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:26.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj40-p3 td/osd-recovery-prio/data 2023-11-02T09:33:26.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:26.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj41-p3 td/osd-recovery-prio/data 2023-11-02T09:33:27.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:27.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj42-p3 td/osd-recovery-prio/data 2023-11-02T09:33:27.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:27.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj43-p3 td/osd-recovery-prio/data 2023-11-02T09:33:28.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:28.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj44-p3 td/osd-recovery-prio/data 2023-11-02T09:33:28.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:28.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj45-p3 td/osd-recovery-prio/data 2023-11-02T09:33:28.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:28.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj46-p3 td/osd-recovery-prio/data 2023-11-02T09:33:29.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:29.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj47-p3 td/osd-recovery-prio/data 2023-11-02T09:33:29.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:29.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj48-p3 td/osd-recovery-prio/data 2023-11-02T09:33:29.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:29.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj49-p3 td/osd-recovery-prio/data 2023-11-02T09:33:30.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:30.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj50-p3 td/osd-recovery-prio/data 2023-11-02T09:33:30.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:30.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj51-p3 td/osd-recovery-prio/data 2023-11-02T09:33:30.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:30.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj52-p3 td/osd-recovery-prio/data 2023-11-02T09:33:31.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:31.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj53-p3 td/osd-recovery-prio/data 2023-11-02T09:33:31.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:31.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj54-p3 td/osd-recovery-prio/data 2023-11-02T09:33:31.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:31.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj55-p3 td/osd-recovery-prio/data 2023-11-02T09:33:32.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:32.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj56-p3 td/osd-recovery-prio/data 2023-11-02T09:33:32.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:32.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj57-p3 td/osd-recovery-prio/data 2023-11-02T09:33:33.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:33.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj58-p3 td/osd-recovery-prio/data 2023-11-02T09:33:33.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:33.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj59-p3 td/osd-recovery-prio/data 2023-11-02T09:33:33.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:33.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj60-p3 td/osd-recovery-prio/data 2023-11-02T09:33:34.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:34.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj61-p3 td/osd-recovery-prio/data 2023-11-02T09:33:34.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:34.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj62-p3 td/osd-recovery-prio/data 2023-11-02T09:33:34.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:34.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj63-p3 td/osd-recovery-prio/data 2023-11-02T09:33:35.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:35.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj64-p3 td/osd-recovery-prio/data 2023-11-02T09:33:35.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:35.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj65-p3 td/osd-recovery-prio/data 2023-11-02T09:33:36.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:36.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj66-p3 td/osd-recovery-prio/data 2023-11-02T09:33:36.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:36.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj67-p3 td/osd-recovery-prio/data 2023-11-02T09:33:36.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:36.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj68-p3 td/osd-recovery-prio/data 2023-11-02T09:33:37.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:37.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj69-p3 td/osd-recovery-prio/data 2023-11-02T09:33:37.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:37.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj70-p3 td/osd-recovery-prio/data 2023-11-02T09:33:37.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:37.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj71-p3 td/osd-recovery-prio/data 2023-11-02T09:33:38.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:38.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj72-p3 td/osd-recovery-prio/data 2023-11-02T09:33:38.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:38.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj73-p3 td/osd-recovery-prio/data 2023-11-02T09:33:38.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:38.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj74-p3 td/osd-recovery-prio/data 2023-11-02T09:33:39.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:39.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj75-p3 td/osd-recovery-prio/data 2023-11-02T09:33:39.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:39.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj76-p3 td/osd-recovery-prio/data 2023-11-02T09:33:40.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:40.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj77-p3 td/osd-recovery-prio/data 2023-11-02T09:33:40.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:40.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj78-p3 td/osd-recovery-prio/data 2023-11-02T09:33:40.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:40.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj79-p3 td/osd-recovery-prio/data 2023-11-02T09:33:41.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:41.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj80-p3 td/osd-recovery-prio/data 2023-11-02T09:33:41.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:41.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj81-p3 td/osd-recovery-prio/data 2023-11-02T09:33:41.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:41.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj82-p3 td/osd-recovery-prio/data 2023-11-02T09:33:42.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:42.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj83-p3 td/osd-recovery-prio/data 2023-11-02T09:33:42.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:42.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj84-p3 td/osd-recovery-prio/data 2023-11-02T09:33:42.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:42.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj85-p3 td/osd-recovery-prio/data 2023-11-02T09:33:43.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:43.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj86-p3 td/osd-recovery-prio/data 2023-11-02T09:33:43.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:43.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj87-p3 td/osd-recovery-prio/data 2023-11-02T09:33:43.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:43.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj88-p3 td/osd-recovery-prio/data 2023-11-02T09:33:44.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:44.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj89-p3 td/osd-recovery-prio/data 2023-11-02T09:33:44.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:44.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj90-p3 td/osd-recovery-prio/data 2023-11-02T09:33:45.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:45.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj91-p3 td/osd-recovery-prio/data 2023-11-02T09:33:45.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:45.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj92-p3 td/osd-recovery-prio/data 2023-11-02T09:33:45.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:45.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj93-p3 td/osd-recovery-prio/data 2023-11-02T09:33:46.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:46.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj94-p3 td/osd-recovery-prio/data 2023-11-02T09:33:46.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:46.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj95-p3 td/osd-recovery-prio/data 2023-11-02T09:33:46.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:46.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj96-p3 td/osd-recovery-prio/data 2023-11-02T09:33:47.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:47.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj97-p3 td/osd-recovery-prio/data 2023-11-02T09:33:47.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:47.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj98-p3 td/osd-recovery-prio/data 2023-11-02T09:33:47.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:47.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj99-p3 td/osd-recovery-prio/data 2023-11-02T09:33:48.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:48.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj100-p3 td/osd-recovery-prio/data 2023-11-02T09:33:48.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:48.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj101-p3 td/osd-recovery-prio/data 2023-11-02T09:33:49.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:49.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj102-p3 td/osd-recovery-prio/data 2023-11-02T09:33:49.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:49.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj103-p3 td/osd-recovery-prio/data 2023-11-02T09:33:49.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:49.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj104-p3 td/osd-recovery-prio/data 2023-11-02T09:33:50.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:50.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj105-p3 td/osd-recovery-prio/data 2023-11-02T09:33:50.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:50.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj106-p3 td/osd-recovery-prio/data 2023-11-02T09:33:50.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:50.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj107-p3 td/osd-recovery-prio/data 2023-11-02T09:33:51.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:51.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj108-p3 td/osd-recovery-prio/data 2023-11-02T09:33:51.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:51.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj109-p3 td/osd-recovery-prio/data 2023-11-02T09:33:51.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:51.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj110-p3 td/osd-recovery-prio/data 2023-11-02T09:33:52.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:52.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj111-p3 td/osd-recovery-prio/data 2023-11-02T09:33:52.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:52.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj112-p3 td/osd-recovery-prio/data 2023-11-02T09:33:53.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:53.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj113-p3 td/osd-recovery-prio/data 2023-11-02T09:33:53.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:53.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj114-p3 td/osd-recovery-prio/data 2023-11-02T09:33:53.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:53.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj115-p3 td/osd-recovery-prio/data 2023-11-02T09:33:54.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:54.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj116-p3 td/osd-recovery-prio/data 2023-11-02T09:33:54.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:54.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj117-p3 td/osd-recovery-prio/data 2023-11-02T09:33:54.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:54.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj118-p3 td/osd-recovery-prio/data 2023-11-02T09:33:55.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:55.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj119-p3 td/osd-recovery-prio/data 2023-11-02T09:33:55.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:55.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj120-p3 td/osd-recovery-prio/data 2023-11-02T09:33:55.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:55.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj121-p3 td/osd-recovery-prio/data 2023-11-02T09:33:56.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:56.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj122-p3 td/osd-recovery-prio/data 2023-11-02T09:33:56.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:56.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj123-p3 td/osd-recovery-prio/data 2023-11-02T09:33:56.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:56.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj124-p3 td/osd-recovery-prio/data 2023-11-02T09:33:57.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:57.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj125-p3 td/osd-recovery-prio/data 2023-11-02T09:33:57.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:57.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj126-p3 td/osd-recovery-prio/data 2023-11-02T09:33:57.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:57.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj127-p3 td/osd-recovery-prio/data 2023-11-02T09:33:58.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:58.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj128-p3 td/osd-recovery-prio/data 2023-11-02T09:33:58.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:58.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj129-p3 td/osd-recovery-prio/data 2023-11-02T09:33:59.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:59.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj130-p3 td/osd-recovery-prio/data 2023-11-02T09:33:59.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:59.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj131-p3 td/osd-recovery-prio/data 2023-11-02T09:33:59.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:33:59.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj132-p3 td/osd-recovery-prio/data 2023-11-02T09:34:00.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:00.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj133-p3 td/osd-recovery-prio/data 2023-11-02T09:34:00.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:00.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj134-p3 td/osd-recovery-prio/data 2023-11-02T09:34:00.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:00.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj135-p3 td/osd-recovery-prio/data 2023-11-02T09:34:01.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:01.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj136-p3 td/osd-recovery-prio/data 2023-11-02T09:34:01.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:01.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj137-p3 td/osd-recovery-prio/data 2023-11-02T09:34:01.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:01.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj138-p3 td/osd-recovery-prio/data 2023-11-02T09:34:02.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:02.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj139-p3 td/osd-recovery-prio/data 2023-11-02T09:34:02.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:02.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj140-p3 td/osd-recovery-prio/data 2023-11-02T09:34:03.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:03.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj141-p3 td/osd-recovery-prio/data 2023-11-02T09:34:03.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:03.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj142-p3 td/osd-recovery-prio/data 2023-11-02T09:34:03.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:03.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj143-p3 td/osd-recovery-prio/data 2023-11-02T09:34:04.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:04.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj144-p3 td/osd-recovery-prio/data 2023-11-02T09:34:04.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:04.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj145-p3 td/osd-recovery-prio/data 2023-11-02T09:34:05.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:05.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj146-p3 td/osd-recovery-prio/data 2023-11-02T09:34:05.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:05.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj147-p3 td/osd-recovery-prio/data 2023-11-02T09:34:05.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:05.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj148-p3 td/osd-recovery-prio/data 2023-11-02T09:34:06.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:06.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj149-p3 td/osd-recovery-prio/data 2023-11-02T09:34:06.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:06.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj150-p3 td/osd-recovery-prio/data 2023-11-02T09:34:06.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:06.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj151-p3 td/osd-recovery-prio/data 2023-11-02T09:34:07.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:07.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj152-p3 td/osd-recovery-prio/data 2023-11-02T09:34:07.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:07.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj153-p3 td/osd-recovery-prio/data 2023-11-02T09:34:08.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:08.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj154-p3 td/osd-recovery-prio/data 2023-11-02T09:34:08.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:08.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj155-p3 td/osd-recovery-prio/data 2023-11-02T09:34:08.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:08.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj156-p3 td/osd-recovery-prio/data 2023-11-02T09:34:09.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:09.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj157-p3 td/osd-recovery-prio/data 2023-11-02T09:34:09.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:09.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj158-p3 td/osd-recovery-prio/data 2023-11-02T09:34:09.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:09.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj159-p3 td/osd-recovery-prio/data 2023-11-02T09:34:10.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:10.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj160-p3 td/osd-recovery-prio/data 2023-11-02T09:34:10.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:10.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj161-p3 td/osd-recovery-prio/data 2023-11-02T09:34:10.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:10.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj162-p3 td/osd-recovery-prio/data 2023-11-02T09:34:11.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:11.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj163-p3 td/osd-recovery-prio/data 2023-11-02T09:34:11.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:11.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj164-p3 td/osd-recovery-prio/data 2023-11-02T09:34:12.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:12.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj165-p3 td/osd-recovery-prio/data 2023-11-02T09:34:12.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:12.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj166-p3 td/osd-recovery-prio/data 2023-11-02T09:34:12.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:12.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj167-p3 td/osd-recovery-prio/data 2023-11-02T09:34:13.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:13.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj168-p3 td/osd-recovery-prio/data 2023-11-02T09:34:13.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:13.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj169-p3 td/osd-recovery-prio/data 2023-11-02T09:34:13.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:13.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj170-p3 td/osd-recovery-prio/data 2023-11-02T09:34:14.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:14.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj171-p3 td/osd-recovery-prio/data 2023-11-02T09:34:14.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:14.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj172-p3 td/osd-recovery-prio/data 2023-11-02T09:34:15.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:15.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj173-p3 td/osd-recovery-prio/data 2023-11-02T09:34:15.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:15.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj174-p3 td/osd-recovery-prio/data 2023-11-02T09:34:15.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:15.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj175-p3 td/osd-recovery-prio/data 2023-11-02T09:34:16.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:16.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj176-p3 td/osd-recovery-prio/data 2023-11-02T09:34:16.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:16.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj177-p3 td/osd-recovery-prio/data 2023-11-02T09:34:16.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:16.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj178-p3 td/osd-recovery-prio/data 2023-11-02T09:34:17.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:17.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj179-p3 td/osd-recovery-prio/data 2023-11-02T09:34:17.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:17.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj180-p3 td/osd-recovery-prio/data 2023-11-02T09:34:17.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:17.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj181-p3 td/osd-recovery-prio/data 2023-11-02T09:34:18.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:18.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj182-p3 td/osd-recovery-prio/data 2023-11-02T09:34:18.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:18.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj183-p3 td/osd-recovery-prio/data 2023-11-02T09:34:19.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:19.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj184-p3 td/osd-recovery-prio/data 2023-11-02T09:34:19.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:19.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj185-p3 td/osd-recovery-prio/data 2023-11-02T09:34:19.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:19.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj186-p3 td/osd-recovery-prio/data 2023-11-02T09:34:20.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:20.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj187-p3 td/osd-recovery-prio/data 2023-11-02T09:34:20.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:20.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj188-p3 td/osd-recovery-prio/data 2023-11-02T09:34:20.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:20.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj189-p3 td/osd-recovery-prio/data 2023-11-02T09:34:21.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:21.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj190-p3 td/osd-recovery-prio/data 2023-11-02T09:34:21.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:21.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj191-p3 td/osd-recovery-prio/data 2023-11-02T09:34:21.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:21.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj192-p3 td/osd-recovery-prio/data 2023-11-02T09:34:22.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:22.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj193-p3 td/osd-recovery-prio/data 2023-11-02T09:34:22.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:22.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj194-p3 td/osd-recovery-prio/data 2023-11-02T09:34:23.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:23.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj195-p3 td/osd-recovery-prio/data 2023-11-02T09:34:23.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:23.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj196-p3 td/osd-recovery-prio/data 2023-11-02T09:34:23.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:23.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj197-p3 td/osd-recovery-prio/data 2023-11-02T09:34:24.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:24.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj198-p3 td/osd-recovery-prio/data 2023-11-02T09:34:24.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:24.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj199-p3 td/osd-recovery-prio/data 2023-11-02T09:34:25.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2023-11-02T09:34:25.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj200-p3 td/osd-recovery-prio/data 2023-11-02T09:34:25.426 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 3 + 1 2023-11-02T09:34:25.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=4 2023-11-02T09:34:25.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: get_not_primary test1 obj1-p1 2023-11-02T09:34:25.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test1 2023-11-02T09:34:25.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1-p1 2023-11-02T09:34:25.429 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test1 obj1-p1 2023-11-02T09:34:25.430 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test1 2023-11-02T09:34:25.430 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1-p1 2023-11-02T09:34:25.430 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test1 obj1-p1 2023-11-02T09:34:25.430 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2023-11-02T09:34:25.845 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2023-11-02T09:34:25.845 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test1 obj1-p1 2023-11-02T09:34:25.845 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2023-11-02T09:34:26.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: local otherosd=0 2023-11-02T09:34:26.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:148: TEST_recovery_priority: ceph pg dump pgs 2023-11-02T09:34:26.614 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T09:34:26.615 INFO:tasks.workunit.client.0.smithi116.stdout:7.0 189 0 0 0 0 1981808640 0 0 567 0 567 active+clean 2023-11-02T09:29:43.665253+0000 91'567 91:620 [1] 1 [1] 1 0'0 2023-11-02T09:29:12.218256+0000 0'0 2023-11-02T09:29:12.218256+0000 0 0 periodic scrub scheduled @ 2023-11-03T19:22:59.855705+0000 0 0 2023-11-02T09:34:26.615 INFO:tasks.workunit.client.0.smithi116.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2023-11-02T09:29:42.651024+0000 91'600 91:663 [1] 1 [1] 1 0'0 2023-11-02T09:29:01.559968+0000 0'0 2023-11-02T09:29:01.559968+0000 0 0 periodic scrub scheduled @ 2023-11-03T20:20:04.372268+0000 0 0 2023-11-02T09:34:26.615 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2023-11-02T09:28:59.632794+0000 91'600 91:672 [1,0] 1 [1,0] 1 0'0 2023-11-02T09:28:56.349957+0000 0'0 2023-11-02T09:28:56.349957+0000 0 0 periodic scrub scheduled @ 2023-11-03T10:48:41.358143+0000 0 0 2023-11-02T09:34:26.615 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T09:34:26.615 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T09:34:26.616 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T09:34:26.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:149: TEST_recovery_priority: ERRORS=0 2023-11-02T09:34:26.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:151: TEST_recovery_priority: ceph osd set norecover 2023-11-02T09:34:27.412 INFO:tasks.workunit.client.0.smithi116.stderr:norecover is set 2023-11-02T09:34:27.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:152: TEST_recovery_priority: ceph osd set noout 2023-11-02T09:34:28.241 INFO:tasks.workunit.client.0.smithi116.stderr:noout is set 2023-11-02T09:34:28.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:156: TEST_recovery_priority: ceph osd pool set test7 size 2 2023-11-02T09:34:29.197 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 7 size to 2 2023-11-02T09:34:29.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:157: TEST_recovery_priority: sleep 2 2023-11-02T09:34:31.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: get_asok_path osd.1 2023-11-02T09:34:31.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-11-02T09:34:31.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-11-02T09:34:31.213 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:34:31.213 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:34:31.213 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:34:31.214 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.261297/ceph-osd.1.asok 2023-11-02T09:34:31.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: CEPH_ARGS= 2023-11-02T09:34:31.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:34:31.299 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:34:31.299 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:34:31.299 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:34:31.299 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:34:31.300 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:34:31.300 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:34:31.300 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:34:31.300 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:34:31.301 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:34:31.302 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:34:31.302 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:34:31.302 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:34:31.302 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:34:31.302 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:34:31.303 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:34:31.303 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:34:31.303 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:34:31.303 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:34:31.303 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:34:31.303 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:34:31.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: seq 1 10 2023-11-02T09:34:31.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2023-11-02T09:34:31.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2023-11-02T09:34:31.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2023-11-02T09:34:31.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:166: TEST_recovery_priority: '[' 1 = 10 ']' 2023-11-02T09:34:31.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: sleep 2 2023-11-02T09:34:33.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2023-11-02T09:34:33.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2023-11-02T09:34:33.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2023-11-02T09:34:34.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:164: TEST_recovery_priority: break 2023-11-02T09:34:34.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:172: TEST_recovery_priority: flush_pg_stats 2023-11-02T09:34:34.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T09:34:34.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T09:34:34.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T09:34:34.461 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T09:34:34.461 INFO:tasks.workunit.client.0.smithi116.stderr:2 2023-11-02T09:34:34.461 INFO:tasks.workunit.client.0.smithi116.stderr:3 2023-11-02T09:34:34.461 INFO:tasks.workunit.client.0.smithi116.stderr:4' 2023-11-02T09:34:34.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T09:34:34.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:34:34.462 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T09:34:34.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836563 2023-11-02T09:34:34.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836563 2023-11-02T09:34:34.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836563' 2023-11-02T09:34:34.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:34:34.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T09:34:34.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673041 2023-11-02T09:34:34.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673041 2023-11-02T09:34:34.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836563 1-42949673041' 2023-11-02T09:34:34.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:34:34.705 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T09:34:34.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509519 2023-11-02T09:34:34.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509519 2023-11-02T09:34:34.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836563 1-42949673041 2-64424509519' 2023-11-02T09:34:34.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:34:34.819 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-11-02T09:34:34.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345997 2023-11-02T09:34:34.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345997 2023-11-02T09:34:34.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836563 1-42949673041 2-64424509519 3-85899345997' 2023-11-02T09:34:34.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:34:34.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-11-02T09:34:35.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182474 2023-11-02T09:34:35.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182474 2023-11-02T09:34:35.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836563 1-42949673041 2-64424509519 3-85899345997 4-107374182474' 2023-11-02T09:34:35.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:34:35.047 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836563 2023-11-02T09:34:35.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:34:35.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T09:34:35.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836563 2023-11-02T09:34:35.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:34:35.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836563 2023-11-02T09:34:35.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836563' 2023-11-02T09:34:35.049 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836563 2023-11-02T09:34:35.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:34:35.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836561 -lt 21474836563 2023-11-02T09:34:35.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T09:34:36.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T09:34:36.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:34:36.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836564 -lt 21474836563 2023-11-02T09:34:36.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:34:36.815 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673041 2023-11-02T09:34:36.816 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:34:36.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T09:34:36.817 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673041 2023-11-02T09:34:36.817 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:34:36.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673041 2023-11-02T09:34:36.818 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949673041 2023-11-02T09:34:36.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673041' 2023-11-02T09:34:36.818 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T09:34:37.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673041 -lt 42949673041 2023-11-02T09:34:37.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:34:37.234 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509519 2023-11-02T09:34:37.234 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:34:37.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T09:34:37.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509519 2023-11-02T09:34:37.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:34:37.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509519 2023-11-02T09:34:37.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509519' 2023-11-02T09:34:37.236 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509519 2023-11-02T09:34:37.237 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T09:34:37.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509519 -lt 64424509519 2023-11-02T09:34:37.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:34:37.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345997 2023-11-02T09:34:37.607 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:34:37.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2023-11-02T09:34:37.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345997 2023-11-02T09:34:37.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:34:37.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345997 2023-11-02T09:34:37.609 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 85899345997 2023-11-02T09:34:37.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345997' 2023-11-02T09:34:37.609 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2023-11-02T09:34:38.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345997 -lt 85899345997 2023-11-02T09:34:38.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:34:38.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182474 2023-11-02T09:34:38.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:34:38.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2023-11-02T09:34:38.021 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182474 2023-11-02T09:34:38.021 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:34:38.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182474 2023-11-02T09:34:38.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182474' 2023-11-02T09:34:38.022 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.4 seq 107374182474 2023-11-02T09:34:38.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2023-11-02T09:34:38.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182475 -lt 107374182474 2023-11-02T09:34:38.429 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: get_asok_path osd.1 2023-11-02T09:34:38.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-11-02T09:34:38.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-11-02T09:34:38.430 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:34:38.430 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:34:38.430 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:34:38.431 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.261297/ceph-osd.1.asok 2023-11-02T09:34:38.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: CEPH_ARGS= 2023-11-02T09:34:38.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:34:38.508 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:34:38.508 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:34:38.508 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:34:38.508 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:34:38.509 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:34:38.510 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:34:38.510 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:34:38.510 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:34:38.510 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:34:38.511 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:34:38.511 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:34:38.511 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:34:38.511 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:34:38.511 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:34:38.512 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:34:38.512 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:34:38.512 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:34:38.512 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:34:38.512 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:34:38.513 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:34:38.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: ceph osd out osd.0 2023-11-02T09:34:39.022 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 is already out. 2023-11-02T09:34:39.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:176: TEST_recovery_priority: sleep 2 2023-11-02T09:34:41.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:177: TEST_recovery_priority: flush_pg_stats 2023-11-02T09:34:41.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T09:34:41.036 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T09:34:41.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T09:34:41.421 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T09:34:41.421 INFO:tasks.workunit.client.0.smithi116.stderr:2 2023-11-02T09:34:41.421 INFO:tasks.workunit.client.0.smithi116.stderr:3 2023-11-02T09:34:41.422 INFO:tasks.workunit.client.0.smithi116.stderr:4' 2023-11-02T09:34:41.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T09:34:41.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:34:41.422 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T09:34:41.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836567 2023-11-02T09:34:41.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836567 2023-11-02T09:34:41.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836567' 2023-11-02T09:34:41.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:34:41.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T09:34:41.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673045 2023-11-02T09:34:41.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673045 2023-11-02T09:34:41.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836567 1-42949673045' 2023-11-02T09:34:41.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:34:41.698 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T09:34:41.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509522 2023-11-02T09:34:41.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509522 2023-11-02T09:34:41.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836567 1-42949673045 2-64424509522' 2023-11-02T09:34:41.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:34:41.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-11-02T09:34:41.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899346000 2023-11-02T09:34:41.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899346000 2023-11-02T09:34:41.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836567 1-42949673045 2-64424509522 3-85899346000' 2023-11-02T09:34:41.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:34:41.918 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-11-02T09:34:42.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182478 2023-11-02T09:34:42.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182478 2023-11-02T09:34:42.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836567 1-42949673045 2-64424509522 3-85899346000 4-107374182478' 2023-11-02T09:34:42.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:34:42.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836567 2023-11-02T09:34:42.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:34:42.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T09:34:42.032 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836567 2023-11-02T09:34:42.032 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:34:42.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836567 2023-11-02T09:34:42.033 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836567 2023-11-02T09:34:42.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836567' 2023-11-02T09:34:42.034 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:34:42.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 0 -lt 21474836567 2023-11-02T09:34:42.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T09:34:43.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T09:34:43.405 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:34:43.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836567 -lt 21474836567 2023-11-02T09:34:43.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:34:43.786 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673045 2023-11-02T09:34:43.786 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:34:43.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T09:34:43.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673045 2023-11-02T09:34:43.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:34:43.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673045 2023-11-02T09:34:43.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673045' 2023-11-02T09:34:43.791 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949673045 2023-11-02T09:34:43.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T09:34:44.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673045 -lt 42949673045 2023-11-02T09:34:44.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:34:44.162 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509522 2023-11-02T09:34:44.162 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:34:44.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T09:34:44.163 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509522 2023-11-02T09:34:44.163 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:34:44.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509522 2023-11-02T09:34:44.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509522' 2023-11-02T09:34:44.164 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509522 2023-11-02T09:34:44.166 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T09:34:44.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509523 -lt 64424509522 2023-11-02T09:34:44.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:34:44.614 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899346000 2023-11-02T09:34:44.614 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:34:44.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2023-11-02T09:34:44.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899346000 2023-11-02T09:34:44.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:34:44.616 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 85899346000 2023-11-02T09:34:44.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899346000 2023-11-02T09:34:44.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899346000' 2023-11-02T09:34:44.618 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2023-11-02T09:34:45.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899346001 -lt 85899346000 2023-11-02T09:34:45.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:34:45.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182478 2023-11-02T09:34:45.074 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:34:45.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2023-11-02T09:34:45.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182478 2023-11-02T09:34:45.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:34:45.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182478 2023-11-02T09:34:45.076 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.4 seq 107374182478 2023-11-02T09:34:45.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182478' 2023-11-02T09:34:45.076 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2023-11-02T09:34:45.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182478 -lt 107374182478 2023-11-02T09:34:45.447 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: get_asok_path osd.1 2023-11-02T09:34:45.447 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-11-02T09:34:45.448 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-11-02T09:34:45.448 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:34:45.448 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:34:45.448 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:34:45.448 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.261297/ceph-osd.1.asok 2023-11-02T09:34:45.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: CEPH_ARGS= 2023-11-02T09:34:45.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:34:45.526 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:34:45.526 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:34:45.526 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:34:45.526 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:34:45.527 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:34:45.527 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:34:45.527 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:34:45.527 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:34:45.527 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:34:45.528 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:34:45.528 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:34:45.528 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:34:45.528 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:34:45.528 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:34:45.529 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:34:45.529 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:34:45.529 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:34:45.529 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:34:45.529 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:34:45.530 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:34:45.530 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:34:45.530 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:34:45.530 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:34:45.530 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:34:45.531 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:34:45.531 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:34:45.531 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:34:45.531 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:34:45.531 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:34:45.532 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:34:45.532 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:34:45.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:179: TEST_recovery_priority: ceph pg dump pgs 2023-11-02T09:34:45.904 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T09:34:45.904 INFO:tasks.workunit.client.0.smithi116.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+forced_recovery+undersized+degraded+remapped 2023-11-02T09:34:33.938440+0000 91'600 103:672 [1,4] 1 [1] 1 0'0 2023-11-02T09:29:12.218256+0000 0'0 2023-11-02T09:29:12.218256+0000 0 0 periodic scrub scheduled @ 2023-11-03T13:24:28.990958+0000 0 0 2023-11-02T09:34:45.904 INFO:tasks.workunit.client.0.smithi116.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2023-11-02T09:29:42.651024+0000 91'600 103:673 [1] 1 [1] 1 0'0 2023-11-02T09:29:01.559968+0000 0'0 2023-11-02T09:29:01.559968+0000 0 0 periodic scrub scheduled @ 2023-11-03T20:20:04.372268+0000 0 0 2023-11-02T09:34:45.905 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2023-11-02T09:34:40.408648+0000 91'600 104:689 [1,2] 1 [1] 1 0'0 2023-11-02T09:28:56.349957+0000 0'0 2023-11-02T09:28:56.349957+0000 0 0 periodic scrub scheduled @ 2023-11-03T20:55:41.402519+0000 0 0 2023-11-02T09:34:45.905 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T09:34:45.905 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T09:34:45.905 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T09:34:45.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: ceph osd pool set test3 size 2 2023-11-02T09:34:46.674 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 3 size to 2 2023-11-02T09:34:46.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:182: TEST_recovery_priority: sleep 2 2023-11-02T09:34:48.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:183: TEST_recovery_priority: flush_pg_stats 2023-11-02T09:34:48.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T09:34:48.687 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T09:34:49.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T09:34:49.102 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T09:34:49.102 INFO:tasks.workunit.client.0.smithi116.stderr:2 2023-11-02T09:34:49.103 INFO:tasks.workunit.client.0.smithi116.stderr:3 2023-11-02T09:34:49.103 INFO:tasks.workunit.client.0.smithi116.stderr:4' 2023-11-02T09:34:49.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T09:34:49.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:34:49.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T09:34:49.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836570 2023-11-02T09:34:49.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836570 2023-11-02T09:34:49.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836570' 2023-11-02T09:34:49.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:34:49.217 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T09:34:49.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673048 2023-11-02T09:34:49.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673048 2023-11-02T09:34:49.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836570 1-42949673048' 2023-11-02T09:34:49.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:34:49.329 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T09:34:49.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509526 2023-11-02T09:34:49.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509526 2023-11-02T09:34:49.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836570 1-42949673048 2-64424509526' 2023-11-02T09:34:49.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:34:49.442 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-11-02T09:34:49.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899346004 2023-11-02T09:34:49.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899346004 2023-11-02T09:34:49.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836570 1-42949673048 2-64424509526 3-85899346004' 2023-11-02T09:34:49.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:34:49.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-11-02T09:34:49.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182481 2023-11-02T09:34:49.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182481 2023-11-02T09:34:49.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836570 1-42949673048 2-64424509526 3-85899346004 4-107374182481' 2023-11-02T09:34:49.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:34:49.669 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836570 2023-11-02T09:34:49.669 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:34:49.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T09:34:49.669 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836570 2023-11-02T09:34:49.669 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:34:49.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836570 2023-11-02T09:34:49.670 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836570 2023-11-02T09:34:49.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836570' 2023-11-02T09:34:49.680 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:34:50.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836568 -lt 21474836570 2023-11-02T09:34:50.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T09:34:51.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T09:34:51.080 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:34:51.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836570 -lt 21474836570 2023-11-02T09:34:51.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:34:51.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673048 2023-11-02T09:34:51.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:34:51.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T09:34:51.465 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673048 2023-11-02T09:34:51.465 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:34:51.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673048 2023-11-02T09:34:51.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673048' 2023-11-02T09:34:51.466 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949673048 2023-11-02T09:34:51.466 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T09:34:51.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673048 -lt 42949673048 2023-11-02T09:34:51.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:34:51.874 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509526 2023-11-02T09:34:51.874 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:34:51.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T09:34:51.875 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509526 2023-11-02T09:34:51.875 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:34:51.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509526 2023-11-02T09:34:51.876 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509526 2023-11-02T09:34:51.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509526' 2023-11-02T09:34:51.877 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T09:34:52.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509526 -lt 64424509526 2023-11-02T09:34:52.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:34:52.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899346004 2023-11-02T09:34:52.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:34:52.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2023-11-02T09:34:52.256 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899346004 2023-11-02T09:34:52.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:34:52.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899346004 2023-11-02T09:34:52.257 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 85899346004 2023-11-02T09:34:52.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899346004' 2023-11-02T09:34:52.258 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2023-11-02T09:34:52.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899346004 -lt 85899346004 2023-11-02T09:34:52.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:34:52.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182481 2023-11-02T09:34:52.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:34:52.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2023-11-02T09:34:52.647 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182481 2023-11-02T09:34:52.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:34:52.648 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.4 seq 107374182481 2023-11-02T09:34:52.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182481 2023-11-02T09:34:52.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182481' 2023-11-02T09:34:52.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2023-11-02T09:34:53.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182482 -lt 107374182481 2023-11-02T09:34:53.095 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: get_asok_path osd.1 2023-11-02T09:34:53.095 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-11-02T09:34:53.095 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-11-02T09:34:53.096 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:34:53.096 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:34:53.096 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:34:53.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.261297/ceph-osd.1.asok 2023-11-02T09:34:53.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: CEPH_ARGS= 2023-11-02T09:34:53.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:34:53.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: cat td/osd-recovery-prio/out 2023-11-02T09:34:53.179 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:34:53.180 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:34:53.180 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:34:53.180 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:34:53.180 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:34:53.180 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:34:53.181 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:34:53.181 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:34:53.181 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:34:53.181 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:34:53.181 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:34:53.182 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:34:53.182 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:34:53.182 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:34:53.182 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:34:53.182 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:34:53.183 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:34:53.183 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:34:53.184 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:34:53.184 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:34:53.184 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:34:53.185 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:34:53.185 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:34:53.185 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:34:53.185 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:34:53.185 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:34:53.185 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:34:53.186 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:34:53.186 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:34:53.187 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:34:53.187 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:34:53.187 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:34:53.187 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:34:53.187 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:34:53.188 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:34:53.188 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:34:53.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:186: TEST_recovery_priority: ceph pg dump pgs 2023-11-02T09:34:53.531 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T09:34:53.531 INFO:tasks.workunit.client.0.smithi116.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+forced_recovery+undersized+degraded+remapped 2023-11-02T09:34:33.938440+0000 91'600 107:675 [1,4] 1 [1] 1 0'0 2023-11-02T09:29:12.218256+0000 0'0 2023-11-02T09:29:12.218256+0000 0 0 periodic scrub scheduled @ 2023-11-03T13:24:28.990958+0000 0 0 2023-11-02T09:34:53.531 INFO:tasks.workunit.client.0.smithi116.stdout:3.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2023-11-02T09:34:48.269938+0000 91'600 108:683 [1,2] 1 [1] 1 0'0 2023-11-02T09:29:01.559968+0000 0'0 2023-11-02T09:29:01.559968+0000 0 0 periodic scrub scheduled @ 2023-11-03T09:49:14.207666+0000 0 0 2023-11-02T09:34:53.531 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2023-11-02T09:34:40.408648+0000 91'600 107:692 [1,2] 1 [1] 1 0'0 2023-11-02T09:28:56.349957+0000 0'0 2023-11-02T09:28:56.349957+0000 0 0 periodic scrub scheduled @ 2023-11-03T20:55:41.402519+0000 0 0 2023-11-02T09:34:53.532 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T09:34:53.532 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T09:34:53.532 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T09:34:53.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: cat td/osd-recovery-prio/out 2023-11-02T09:34:53.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2023-11-02T09:34:53.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: PRIO=190 2023-11-02T09:34:53.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:189: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2023-11-02T09:34:53.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: cat td/osd-recovery-prio/out 2023-11-02T09:34:53.570 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2023-11-02T09:34:53.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: eval 'ITEM="7.0"' 2023-11-02T09:34:53.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: ITEM=7.0 2023-11-02T09:34:53.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:197: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2023-11-02T09:34:53.597 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: cat td/osd-recovery-prio/out 2023-11-02T09:34:53.597 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2023-11-02T09:34:53.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: PRIO=255 2023-11-02T09:34:53.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:203: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2023-11-02T09:34:53.623 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: seq 1 10 2023-11-02T09:34:53.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2023-11-02T09:34:53.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:213: TEST_recovery_priority: ceph pg force-recovery 3.0 2023-11-02T09:34:53.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:213: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2023-11-02T09:34:53.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:214: TEST_recovery_priority: break 2023-11-02T09:34:53.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: sleep 2 2023-11-02T09:34:55.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: get_asok_path osd.1 2023-11-02T09:34:55.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-11-02T09:34:55.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-11-02T09:34:55.988 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:34:55.989 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:34:55.989 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:34:55.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.261297/ceph-osd.1.asok 2023-11-02T09:34:55.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: CEPH_ARGS= 2023-11-02T09:34:55.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:34:56.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:224: TEST_recovery_priority: cat td/osd-recovery-prio/out 2023-11-02T09:34:56.084 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:34:56.085 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:34:56.085 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:34:56.085 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:34:56.085 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:34:56.085 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:34:56.086 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:34:56.086 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:34:56.086 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:34:56.086 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:34:56.086 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:34:56.087 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:34:56.087 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:34:56.087 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:34:56.087 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:34:56.087 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:34:56.088 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 255, 2023-11-02T09:34:56.088 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:34:56.088 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:34:56.088 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:34:56.088 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:34:56.089 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:34:56.089 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:34:56.089 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:34:56.089 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:34:56.089 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:34:56.089 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:34:56.090 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:34:56.090 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:34:56.090 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:34:56.090 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:34:56.090 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:34:56.091 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:34:56.091 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:34:56.091 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:34:56.091 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:34:56.091 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:34:56.092 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:34:56.092 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:34:56.092 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:34:56.092 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:34:56.093 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: cat td/osd-recovery-prio/out 2023-11-02T09:34:56.093 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2023-11-02T09:34:56.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: PRIO=255 2023-11-02T09:34:56.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:226: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2023-11-02T09:34:56.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:231: TEST_recovery_priority: flush_pg_stats 2023-11-02T09:34:56.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T09:34:56.118 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T09:34:56.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T09:34:56.504 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T09:34:56.504 INFO:tasks.workunit.client.0.smithi116.stderr:2 2023-11-02T09:34:56.505 INFO:tasks.workunit.client.0.smithi116.stderr:3 2023-11-02T09:34:56.505 INFO:tasks.workunit.client.0.smithi116.stderr:4' 2023-11-02T09:34:56.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T09:34:56.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:34:56.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T09:34:56.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836574 2023-11-02T09:34:56.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836574 2023-11-02T09:34:56.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836574' 2023-11-02T09:34:56.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:34:56.617 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T09:34:56.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673052 2023-11-02T09:34:56.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673052 2023-11-02T09:34:56.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836574 1-42949673052' 2023-11-02T09:34:56.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:34:56.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T09:34:56.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509529 2023-11-02T09:34:56.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509529 2023-11-02T09:34:56.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836574 1-42949673052 2-64424509529' 2023-11-02T09:34:56.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:34:56.863 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-11-02T09:34:56.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899346007 2023-11-02T09:34:56.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899346007 2023-11-02T09:34:56.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836574 1-42949673052 2-64424509529 3-85899346007' 2023-11-02T09:34:56.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:34:56.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-11-02T09:34:57.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182485 2023-11-02T09:34:57.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182485 2023-11-02T09:34:57.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836574 1-42949673052 2-64424509529 3-85899346007 4-107374182485' 2023-11-02T09:34:57.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:34:57.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836574 2023-11-02T09:34:57.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:34:57.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T09:34:57.088 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836574 2023-11-02T09:34:57.088 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:34:57.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836574 2023-11-02T09:34:57.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836574' 2023-11-02T09:34:57.089 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836574 2023-11-02T09:34:57.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:34:57.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836572 -lt 21474836574 2023-11-02T09:34:57.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T09:34:58.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T09:34:58.460 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:34:58.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836574 -lt 21474836574 2023-11-02T09:34:58.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:34:58.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673052 2023-11-02T09:34:58.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:34:58.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T09:34:58.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673052 2023-11-02T09:34:58.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:34:58.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673052 2023-11-02T09:34:58.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2023-11-02T09:34:58.937 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949673052 2023-11-02T09:34:58.937 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T09:34:59.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673052 -lt 42949673052 2023-11-02T09:34:59.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:34:59.344 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509529 2023-11-02T09:34:59.344 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:34:59.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T09:34:59.345 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509529 2023-11-02T09:34:59.345 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:34:59.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509529 2023-11-02T09:34:59.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509529' 2023-11-02T09:34:59.347 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509529 2023-11-02T09:34:59.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T09:34:59.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509530 -lt 64424509529 2023-11-02T09:34:59.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:34:59.761 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899346007 2023-11-02T09:34:59.762 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:34:59.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2023-11-02T09:34:59.763 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899346007 2023-11-02T09:34:59.763 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:34:59.763 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 85899346007 2023-11-02T09:34:59.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899346007 2023-11-02T09:34:59.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899346007' 2023-11-02T09:34:59.764 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2023-11-02T09:35:00.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899346007 -lt 85899346007 2023-11-02T09:35:00.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:35:00.180 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182485 2023-11-02T09:35:00.180 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:35:00.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2023-11-02T09:35:00.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182485 2023-11-02T09:35:00.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:35:00.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182485 2023-11-02T09:35:00.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182485' 2023-11-02T09:35:00.182 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.4 seq 107374182485 2023-11-02T09:35:00.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2023-11-02T09:35:00.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182486 -lt 107374182485 2023-11-02T09:35:00.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: ceph pg cancel-force-recovery 7.0 2023-11-02T09:35:00.919 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-recovery; 2023-11-02T09:35:00.920 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-11-02T09:35:00.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:236: TEST_recovery_priority: sleep 2 2023-11-02T09:35:02.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: get_asok_path osd.1 2023-11-02T09:35:02.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-11-02T09:35:02.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-11-02T09:35:02.935 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:35:02.935 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:35:02.936 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:35:02.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.261297/ceph-osd.1.asok 2023-11-02T09:35:02.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: CEPH_ARGS= 2023-11-02T09:35:02.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:03.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:239: TEST_recovery_priority: cat td/osd-recovery-prio/out 2023-11-02T09:35:03.028 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:03.028 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:03.028 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:03.028 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:03.029 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:03.029 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:03.029 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:03.029 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:03.029 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:03.030 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:03.030 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:03.030 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:03.030 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:03.030 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:03.031 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:03.031 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:03.031 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:03.031 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:03.031 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:03.032 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:03.032 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:03.032 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:03.032 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:03.032 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:03.033 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:03.033 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:03.033 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:03.033 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:03.033 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:03.033 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:03.035 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:03.035 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:03.035 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:03.035 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:03.036 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:03.036 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:03.036 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: cat td/osd-recovery-prio/out 2023-11-02T09:35:03.037 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2023-11-02T09:35:03.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: PRIO=190 2023-11-02T09:35:03.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:241: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2023-11-02T09:35:03.058 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: cat td/osd-recovery-prio/out 2023-11-02T09:35:03.058 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2023-11-02T09:35:03.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: eval 'ITEM="3.0"' 2023-11-02T09:35:03.086 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: ITEM=3.0 2023-11-02T09:35:03.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:248: TEST_recovery_priority: '[' 3.0 '!=' 3.0 ']' 2023-11-02T09:35:03.086 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: cat td/osd-recovery-prio/out 2023-11-02T09:35:03.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2023-11-02T09:35:03.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: PRIO=255 2023-11-02T09:35:03.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:254: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2023-11-02T09:35:03.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:261: TEST_recovery_priority: ceph pg cancel-force-recovery 3.0 2023-11-02T09:35:03.464 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-recovery; 2023-11-02T09:35:03.464 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-11-02T09:35:03.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:262: TEST_recovery_priority: sleep 5 2023-11-02T09:35:08.477 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: get_asok_path osd.1 2023-11-02T09:35:08.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-11-02T09:35:08.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-11-02T09:35:08.479 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:35:08.479 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:35:08.479 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:35:08.480 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.261297/ceph-osd.1.asok 2023-11-02T09:35:08.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: CEPH_ARGS= 2023-11-02T09:35:08.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:08.565 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:08.566 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:08.566 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:08.566 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:08.567 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:08.567 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:08.567 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:08.568 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:08.568 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:08.568 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:08.568 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:08.569 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:08.569 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:08.569 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:08.569 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:08.569 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:08.570 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:08.570 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:08.570 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:08.570 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:08.570 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:08.571 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:08.571 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:08.571 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:08.571 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:08.571 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:08.572 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:08.572 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:08.572 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:08.572 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:08.572 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:08.572 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:08.573 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:08.573 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:08.573 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:08.574 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:08.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:266: TEST_recovery_priority: flush_pg_stats 2023-11-02T09:35:08.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T09:35:08.575 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T09:35:08.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T09:35:08.950 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T09:35:08.950 INFO:tasks.workunit.client.0.smithi116.stderr:2 2023-11-02T09:35:08.950 INFO:tasks.workunit.client.0.smithi116.stderr:3 2023-11-02T09:35:08.950 INFO:tasks.workunit.client.0.smithi116.stderr:4' 2023-11-02T09:35:08.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T09:35:08.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:35:08.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T09:35:09.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836578 2023-11-02T09:35:09.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836578 2023-11-02T09:35:09.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836578' 2023-11-02T09:35:09.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:35:09.063 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T09:35:09.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673056 2023-11-02T09:35:09.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673056 2023-11-02T09:35:09.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836578 1-42949673056' 2023-11-02T09:35:09.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:35:09.176 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T09:35:09.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509534 2023-11-02T09:35:09.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509534 2023-11-02T09:35:09.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836578 1-42949673056 2-64424509534' 2023-11-02T09:35:09.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:35:09.288 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-11-02T09:35:09.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899346012 2023-11-02T09:35:09.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899346012 2023-11-02T09:35:09.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836578 1-42949673056 2-64424509534 3-85899346012' 2023-11-02T09:35:09.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:35:09.402 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-11-02T09:35:09.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182489 2023-11-02T09:35:09.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182489 2023-11-02T09:35:09.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836578 1-42949673056 2-64424509534 3-85899346012 4-107374182489' 2023-11-02T09:35:09.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:35:09.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836578 2023-11-02T09:35:09.517 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:35:09.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T09:35:09.518 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836578 2023-11-02T09:35:09.518 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:35:09.518 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836578 2023-11-02T09:35:09.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836578 2023-11-02T09:35:09.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836578' 2023-11-02T09:35:09.519 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:35:09.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836576 -lt 21474836578 2023-11-02T09:35:09.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T09:35:10.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T09:35:10.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:35:11.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836578 -lt 21474836578 2023-11-02T09:35:11.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:35:11.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673056 2023-11-02T09:35:11.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:35:11.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T09:35:11.316 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673056 2023-11-02T09:35:11.316 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:35:11.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673056 2023-11-02T09:35:11.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673056' 2023-11-02T09:35:11.317 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949673056 2023-11-02T09:35:11.318 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T09:35:11.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673056 -lt 42949673056 2023-11-02T09:35:11.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:35:11.691 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509534 2023-11-02T09:35:11.691 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:35:11.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T09:35:11.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509534 2023-11-02T09:35:11.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:35:11.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509534 2023-11-02T09:35:11.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509534' 2023-11-02T09:35:11.694 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509534 2023-11-02T09:35:11.694 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T09:35:12.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509534 -lt 64424509534 2023-11-02T09:35:12.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:35:12.068 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899346012 2023-11-02T09:35:12.068 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:35:12.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2023-11-02T09:35:12.069 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899346012 2023-11-02T09:35:12.069 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:35:12.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899346012 2023-11-02T09:35:12.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899346012' 2023-11-02T09:35:12.070 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 85899346012 2023-11-02T09:35:12.071 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2023-11-02T09:35:12.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899346012 -lt 85899346012 2023-11-02T09:35:12.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:35:12.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182489 2023-11-02T09:35:12.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:35:12.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2023-11-02T09:35:12.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182489 2023-11-02T09:35:12.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:35:12.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182489 2023-11-02T09:35:12.442 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.4 seq 107374182489 2023-11-02T09:35:12.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182489' 2023-11-02T09:35:12.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2023-11-02T09:35:12.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182490 -lt 107374182489 2023-11-02T09:35:12.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: ceph pg force-recovery 7.0 2023-11-02T09:35:13.183 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg(s) [7.0] on osd.1 to force-recovery; 2023-11-02T09:35:13.183 INFO:tasks.workunit.client.0.smithi116.stderr: 2023-11-02T09:35:13.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:268: TEST_recovery_priority: sleep 2 2023-11-02T09:35:15.198 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: get_asok_path osd.1 2023-11-02T09:35:15.199 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-11-02T09:35:15.199 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-11-02T09:35:15.199 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:35:15.200 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:35:15.200 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:35:15.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.261297/ceph-osd.1.asok 2023-11-02T09:35:15.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: CEPH_ARGS= 2023-11-02T09:35:15.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:15.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:271: TEST_recovery_priority: cat td/osd-recovery-prio/out 2023-11-02T09:35:15.292 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:15.292 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:15.292 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:15.292 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:15.293 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:15.293 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:15.293 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:15.293 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:15.293 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:15.294 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:15.294 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:15.297 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:15.298 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:15.298 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:15.298 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:15.298 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:15.298 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:15.299 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:15.299 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:15.299 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:15.299 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:15.299 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:15.300 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:15.300 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:15.300 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:15.300 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:15.300 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:15.301 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:15.301 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:15.301 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:15.301 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:15.301 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:15.302 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:15.302 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:15.302 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:15.302 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:15.303 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: cat td/osd-recovery-prio/out 2023-11-02T09:35:15.303 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2023-11-02T09:35:15.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: PRIO=190 2023-11-02T09:35:15.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:273: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2023-11-02T09:35:15.322 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: cat td/osd-recovery-prio/out 2023-11-02T09:35:15.322 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2023-11-02T09:35:15.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: eval 'ITEM="7.0"' 2023-11-02T09:35:15.350 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: ITEM=7.0 2023-11-02T09:35:15.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:280: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2023-11-02T09:35:15.350 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: cat td/osd-recovery-prio/out 2023-11-02T09:35:15.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2023-11-02T09:35:15.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: PRIO=255 2023-11-02T09:35:15.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:286: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2023-11-02T09:35:15.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:293: TEST_recovery_priority: ceph osd unset noout 2023-11-02T09:35:16.060 INFO:tasks.workunit.client.0.smithi116.stderr:noout is unset 2023-11-02T09:35:16.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:294: TEST_recovery_priority: ceph osd unset norecover 2023-11-02T09:35:16.730 INFO:tasks.workunit.client.0.smithi116.stderr:norecover is unset 2023-11-02T09:35:16.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:296: TEST_recovery_priority: get_asok_path osd.1 2023-11-02T09:35:16.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-11-02T09:35:16.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-11-02T09:35:16.743 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:35:16.743 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:35:16.744 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:35:16.744 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.261297/ceph-osd.1.asok 2023-11-02T09:35:16.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:296: TEST_recovery_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations' 2023-11-02T09:35:16.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations' 2023-11-02T09:35:16.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T09:35:16.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T09:35:16.745 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-11-02T09:35:16.746 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T09:35:16.746 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T09:35:16.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T09:35:16.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T09:35:16.747 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T09:35:16.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-11-02T09:35:16.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T09:35:16.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T09:35:16.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T09:35:16.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T09:35:16.827 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T09:35:17.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T09:35:17.244 INFO:tasks.workunit.client.0.smithi116.stderr:1 2023-11-02T09:35:17.244 INFO:tasks.workunit.client.0.smithi116.stderr:2 2023-11-02T09:35:17.244 INFO:tasks.workunit.client.0.smithi116.stderr:3 2023-11-02T09:35:17.244 INFO:tasks.workunit.client.0.smithi116.stderr:4' 2023-11-02T09:35:17.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T09:35:17.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:35:17.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T09:35:17.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836582 2023-11-02T09:35:17.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836582 2023-11-02T09:35:17.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836582' 2023-11-02T09:35:17.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:35:17.358 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T09:35:17.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949673060 2023-11-02T09:35:17.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949673060 2023-11-02T09:35:17.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836582 1-42949673060' 2023-11-02T09:35:17.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:35:17.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-11-02T09:35:17.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509537 2023-11-02T09:35:17.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509537 2023-11-02T09:35:17.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836582 1-42949673060 2-64424509537' 2023-11-02T09:35:17.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:35:17.585 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-11-02T09:35:17.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899346015 2023-11-02T09:35:17.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899346015 2023-11-02T09:35:17.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836582 1-42949673060 2-64424509537 3-85899346015' 2023-11-02T09:35:17.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:35:17.698 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-11-02T09:35:17.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182493 2023-11-02T09:35:17.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182493 2023-11-02T09:35:17.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836582 1-42949673060 2-64424509537 3-85899346015 4-107374182493' 2023-11-02T09:35:17.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:35:17.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836582 2023-11-02T09:35:17.812 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:35:17.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T09:35:17.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836582 2023-11-02T09:35:17.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:35:17.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836582 2023-11-02T09:35:17.814 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836582 2023-11-02T09:35:17.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2023-11-02T09:35:17.814 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:35:18.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836580 -lt 21474836582 2023-11-02T09:35:18.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T09:35:19.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T09:35:19.196 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:35:19.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836582 -lt 21474836582 2023-11-02T09:35:19.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:35:19.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949673060 2023-11-02T09:35:19.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:35:19.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T09:35:19.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949673060 2023-11-02T09:35:19.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:35:19.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949673060 2023-11-02T09:35:19.588 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949673060 2023-11-02T09:35:19.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949673060' 2023-11-02T09:35:19.589 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T09:35:20.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949673060 -lt 42949673060 2023-11-02T09:35:20.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:35:20.028 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509537 2023-11-02T09:35:20.028 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:35:20.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-11-02T09:35:20.029 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509537 2023-11-02T09:35:20.030 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:35:20.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509537 2023-11-02T09:35:20.030 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509537 2023-11-02T09:35:20.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509537' 2023-11-02T09:35:20.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-11-02T09:35:20.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509538 -lt 64424509537 2023-11-02T09:35:20.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:35:20.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899346015 2023-11-02T09:35:20.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:35:20.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2023-11-02T09:35:20.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899346015 2023-11-02T09:35:20.413 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:35:20.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899346015 2023-11-02T09:35:20.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899346015' 2023-11-02T09:35:20.414 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 85899346015 2023-11-02T09:35:20.415 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2023-11-02T09:35:20.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899346016 -lt 85899346015 2023-11-02T09:35:20.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:35:20.995 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182493 2023-11-02T09:35:20.995 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:35:20.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2023-11-02T09:35:20.996 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182493 2023-11-02T09:35:20.996 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:35:20.997 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.4 seq 107374182493 2023-11-02T09:35:20.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182493 2023-11-02T09:35:20.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182493' 2023-11-02T09:35:20.998 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2023-11-02T09:35:21.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182494 -lt 107374182493 2023-11-02T09:35:21.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T09:35:21.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:21.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:22.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2023-11-02T09:35:22.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:22.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:22.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:22.141 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:22.141 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:22.141 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:22.141 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:22.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:22.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:22.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:22.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:23.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:23.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2023-11-02T09:35:23.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2023-11-02T09:35:23.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2023-11-02T09:35:23.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:23.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:23.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:23.279 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:23.279 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:23.279 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:23.279 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:23.280 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:23.280 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:23.280 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:23.280 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:23.280 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:23.281 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:23.282 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:23.282 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:23.282 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:23.283 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:23.283 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:23.283 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:23.283 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:23.283 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:23.284 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:23.284 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:23.284 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:23.284 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:23.284 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:23.284 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:23.285 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:23.285 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:23.285 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:23.285 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:23.285 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:23.286 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:23.286 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:23.286 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:23.286 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:23.286 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:23.286 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:23.287 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:23.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:23.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:23.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:23.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:23.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:23.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:23.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:23.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:23.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:23.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:23.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:23.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:23.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:24.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:24.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:24.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:24.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:24.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:24.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:24.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:24.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:24.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:24.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6552351 2023-11-02T09:35:24.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6552351 '!=' null 2023-11-02T09:35:24.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:24.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:24.889 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:24.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:24.968 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:24.969 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:24.969 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:24.969 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:24.969 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:24.969 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:24.970 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:24.970 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:24.970 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:24.970 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:24.970 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:24.971 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:24.971 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:24.971 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:24.971 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:24.971 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:24.972 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:24.972 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:24.972 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:24.972 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:24.972 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:24.973 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:24.973 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:24.973 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:24.973 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:24.973 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:24.973 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:24.974 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:24.974 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:24.974 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:24.974 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:24.974 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:24.975 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:24.975 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:24.975 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:24.975 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:24.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:25.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:25.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:25.077 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:25.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:25.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:25.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:25.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:25.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:25.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:25.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:25.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:25.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:25.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:25.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:25.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:25.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:25.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:25.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:25.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:25.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:25.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:26.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6552351 2023-11-02T09:35:26.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6552351 '!=' null 2023-11-02T09:35:26.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:26.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:26.533 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:26.533 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:26.609 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:26.609 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:26.610 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:26.610 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:26.610 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:26.610 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:26.611 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:26.611 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:26.611 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:26.612 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:26.612 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:26.612 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:26.613 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:26.613 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:26.613 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:26.614 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:26.614 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:26.614 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:26.615 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:26.615 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:26.616 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:26.616 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:26.616 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:26.617 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:26.617 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:26.617 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:26.618 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:26.618 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:26.618 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:26.620 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:26.620 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:26.623 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:26.623 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:26.624 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:26.624 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:26.624 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:26.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:26.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:26.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:26.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:26.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:26.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:26.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:26.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:26.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:27.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:27.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:27.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:27.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:27.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:27.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:27.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:27.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:27.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:27.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:27.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:27.705 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:27.705 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:28.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5308361 2023-11-02T09:35:28.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5308361 '!=' null 2023-11-02T09:35:28.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:28.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:28.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:28.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:28.348 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:28.348 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:28.348 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:28.348 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:28.349 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:28.349 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:28.349 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:28.349 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:28.350 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:28.350 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:28.350 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:28.351 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:28.351 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:28.351 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:28.351 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:28.352 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:28.352 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:28.352 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:28.352 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:28.352 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:28.353 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:28.353 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:28.353 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:28.353 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:28.353 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:28.354 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:28.354 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:28.354 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:28.354 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:28.355 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:28.355 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:28.356 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:28.356 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:28.356 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:28.357 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:28.357 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:28.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:28.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:28.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:28.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:28.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:28.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:28.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:28.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:28.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:28.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:28.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:28.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:28.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:29.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:29.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:29.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:29.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:29.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:29.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:29.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:29.484 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:29.484 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:30.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13509745 2023-11-02T09:35:30.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13509745 '!=' null 2023-11-02T09:35:30.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:30.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:30.018 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:30.019 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:30.094 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:30.095 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:30.095 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:30.095 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:30.095 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:30.095 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:30.096 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:30.096 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:30.096 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:30.096 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:30.096 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:30.097 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:30.097 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:30.097 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:30.097 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:30.097 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:30.098 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:30.098 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:30.098 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:30.098 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:30.098 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:30.100 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:30.100 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:30.100 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:30.100 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:30.101 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:30.101 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:30.101 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:30.101 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:30.101 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:30.101 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:30.102 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:30.102 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:30.102 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:30.102 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:30.102 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:30.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:30.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:30.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:30.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:30.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:30.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:30.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:30.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:30.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:30.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:30.605 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:30.605 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:30.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:31.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:31.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:31.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:31.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:31.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:31.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:31.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:31.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:31.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:31.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13104645 2023-11-02T09:35:31.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13104645 '!=' null 2023-11-02T09:35:31.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:31.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:31.828 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:31.828 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:31.906 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:31.906 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:31.907 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:31.907 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:31.907 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:31.907 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:31.907 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:31.908 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:31.908 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:31.908 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:31.908 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:31.908 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:31.909 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:31.909 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:31.909 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:31.909 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:31.909 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:31.910 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:31.910 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:31.910 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:31.910 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:31.910 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:31.911 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:31.911 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:31.911 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:31.911 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:31.911 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:31.912 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:31.912 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:31.912 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:31.913 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:31.913 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:31.913 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:31.913 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:31.913 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:31.914 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:31.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:32.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:32.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:32.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:32.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:32.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:32.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:32.014 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:32.014 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:32.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:32.381 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:32.381 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:32.382 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:33.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:33.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:33.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:33.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:33.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:33.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:33.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:33.099 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:33.100 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:33.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20967902 2023-11-02T09:35:33.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20967902 '!=' null 2023-11-02T09:35:33.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:33.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:33.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:33.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:33.701 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:33.702 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:33.702 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:33.702 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:33.702 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:33.702 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:33.703 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:33.703 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:33.703 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:33.703 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:33.703 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:33.704 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:33.704 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:33.704 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:33.704 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:33.704 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:33.704 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:33.705 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:33.705 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:33.705 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:33.705 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:33.705 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:33.706 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:33.706 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:33.706 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:33.706 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:33.706 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:33.707 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:33.707 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:33.707 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:33.707 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:33.707 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:33.708 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:33.708 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:33.708 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:33.708 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:33.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:33.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:33.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:33.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:33.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:33.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:33.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:33.812 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:33.812 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:34.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:34.179 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:34.179 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:34.179 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:34.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:34.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:34.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:34.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:34.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:34.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:34.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:34.690 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:34.690 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:35.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599604 2023-11-02T09:35:35.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599604 '!=' null 2023-11-02T09:35:35.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:35.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:35.349 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:35.349 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:35.425 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:35.426 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:35.426 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:35.426 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:35.426 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:35.427 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:35.427 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:35.427 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:35.427 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:35.427 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:35.428 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:35.428 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:35.428 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:35.428 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:35.428 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:35.429 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:35.429 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:35.429 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:35.429 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:35.429 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:35.430 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:35.430 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:35.430 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:35.430 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:35.430 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:35.431 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:35.431 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:35.431 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:35.431 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:35.431 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:35.432 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:35.432 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:35.432 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:35.432 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:35.432 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:35.432 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:35.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:35.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:35.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:35.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:35.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:35.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:35.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:35.536 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:35.536 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:35.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:35.900 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:35.900 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:35.901 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:36.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:36.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:36.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:36.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:36.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:36.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:36.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:36.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:36.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:37.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599619 2023-11-02T09:35:37.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599619 '!=' null 2023-11-02T09:35:37.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:37.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:37.014 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:37.015 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:37.091 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:37.092 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:37.092 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:37.092 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:37.092 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:37.092 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:37.093 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:37.093 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:37.093 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:37.093 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:37.093 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:37.094 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:37.094 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:37.094 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:37.094 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:37.094 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:37.095 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:37.095 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:37.095 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:37.095 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:37.095 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:37.095 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:37.096 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:37.096 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:37.096 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:37.096 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:37.096 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:37.097 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:37.097 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:37.097 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:37.097 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:37.097 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:37.097 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:37.098 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:37.098 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:37.098 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:37.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:37.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:37.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:37.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:37.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:37.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:37.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:37.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:37.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:37.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:37.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:37.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:37.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:38.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:38.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:38.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:38.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:38.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:38.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:38.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:38.080 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:38.080 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:38.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210011 2023-11-02T09:35:38.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210011 '!=' null 2023-11-02T09:35:38.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:38.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:38.886 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:38.886 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:38.965 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:38.965 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:38.965 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:38.965 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:38.966 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:38.966 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:38.966 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:38.966 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:38.966 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:38.967 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:38.967 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:38.967 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:38.967 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:38.967 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:38.967 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:38.968 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:38.968 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:38.968 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:38.968 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:38.968 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:38.969 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:38.969 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:38.969 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:38.969 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:38.969 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:38.969 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:38.970 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:38.970 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:38.970 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:38.970 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:38.970 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:38.971 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:38.972 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:38.972 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:38.972 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:38.972 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:38.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:39.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:39.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:39.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:39.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:39.074 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:39.074 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:39.074 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:39.074 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:39.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:39.500 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:39.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:39.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:40.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:40.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:40.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:40.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:40.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:40.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:40.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:40.010 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:40.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:40.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473449 2023-11-02T09:35:40.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473449 '!=' null 2023-11-02T09:35:40.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:40.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:40.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:40.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:40.684 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:40.685 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:40.685 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:40.685 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:40.685 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:40.685 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:40.686 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:40.686 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:40.686 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:40.686 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:40.687 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:40.687 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:40.687 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:40.687 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:40.687 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:40.688 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:40.688 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:40.688 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:40.688 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:40.688 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:40.689 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:40.689 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:40.689 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:40.689 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:40.689 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:40.690 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:40.690 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:40.690 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:40.690 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:40.691 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:40.691 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:40.691 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:40.691 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:40.691 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:40.692 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:40.692 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:40.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:40.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:40.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:40.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:40.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:40.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:40.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:40.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:40.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:41.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:41.176 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:41.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:41.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:41.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:41.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:41.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:41.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:41.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:41.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:41.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:41.792 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:41.792 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:42.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473449 2023-11-02T09:35:42.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473449 '!=' null 2023-11-02T09:35:42.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:42.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:42.375 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:42.375 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:42.452 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:42.453 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:42.453 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:42.453 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:42.453 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:42.453 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:42.454 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:42.454 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:42.454 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:42.454 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:42.455 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:42.455 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:42.455 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:42.455 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:42.455 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:42.456 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:42.456 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:42.456 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:42.456 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:42.456 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:42.457 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:42.457 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:42.457 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:42.457 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:42.457 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:42.458 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:42.458 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:42.458 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:42.458 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:42.458 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:42.459 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:42.459 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:42.459 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:42.459 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:42.459 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:42.460 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:42.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:42.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:42.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:42.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:42.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:42.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:42.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:42.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:42.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:42.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:42.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:42.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:42.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:43.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:43.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:43.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:43.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:43.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:43.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:43.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:43.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:43.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:44.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336584 2023-11-02T09:35:44.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336584 '!=' null 2023-11-02T09:35:44.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:44.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:44.082 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:44.082 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:44.159 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:44.159 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:44.160 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:44.160 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:44.160 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:44.160 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:44.160 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:44.161 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:44.161 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:44.161 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:44.161 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:44.161 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:44.162 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:44.162 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:44.162 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:44.162 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:44.162 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:44.163 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:44.163 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:44.163 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:44.163 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:44.163 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:44.163 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:44.164 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:44.167 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:44.167 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:44.167 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:44.167 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:44.168 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:44.168 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:44.168 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:44.169 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:44.169 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:44.169 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:44.169 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:44.170 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:44.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:44.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:44.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:44.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:44.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:44.269 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:44.269 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:44.269 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:44.269 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:44.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:44.634 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:44.634 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:44.634 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:45.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:45.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:45.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:45.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:45.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:45.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:45.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:45.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:45.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:45.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473451 2023-11-02T09:35:45.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473451 '!=' null 2023-11-02T09:35:45.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:45.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:45.668 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:45.668 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:45.742 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:45.743 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:45.743 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:45.743 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:45.743 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:45.743 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:45.744 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:45.744 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:45.744 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:45.744 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:45.744 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:45.745 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:45.745 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:45.745 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:45.745 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:45.745 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:45.746 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:45.746 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:45.746 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:45.747 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:45.747 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:45.747 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:45.748 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:45.748 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:45.748 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:45.748 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:45.749 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:45.750 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:45.750 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:45.750 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:45.750 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:45.750 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:45.751 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:45.751 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:45.751 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:45.751 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:45.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:45.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:45.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:45.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:45.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:45.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:45.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:45.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:45.854 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:46.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:46.238 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:46.239 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:46.239 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:46.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:46.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:46.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:46.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:46.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:46.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:46.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:46.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:46.785 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:47.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473305 2023-11-02T09:35:47.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473305 '!=' null 2023-11-02T09:35:47.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:47.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:47.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:47.379 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:47.453 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:47.454 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:47.454 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:47.454 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:47.454 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:47.455 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:47.455 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:47.455 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:47.455 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:47.456 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:47.456 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:47.456 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:47.456 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:47.456 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:47.457 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:47.457 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:47.457 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:47.457 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:47.457 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:47.458 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:47.458 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:47.458 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:47.463 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:47.463 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:47.463 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:47.463 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:47.463 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:47.464 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:47.464 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:47.464 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:47.464 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:47.464 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:47.465 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:47.465 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:47.465 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:47.466 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:47.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:47.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:47.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:47.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:47.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:47.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:47.564 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:47.564 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:47.564 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:47.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:47.931 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:47.931 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:47.931 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:48.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:48.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:48.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:48.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:48.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:48.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:48.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:48.610 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:48.610 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:49.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209868 2023-11-02T09:35:49.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209868 '!=' null 2023-11-02T09:35:49.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:49.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:49.117 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:49.117 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:49.194 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:49.194 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:49.194 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:49.194 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:49.195 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:49.195 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:49.195 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:49.195 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:49.195 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:49.195 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:49.196 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:49.196 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:49.196 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:49.196 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:49.196 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:49.197 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:49.197 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:49.197 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:49.197 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:49.197 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:49.198 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:49.198 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:49.198 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:49.198 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:49.198 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:49.199 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:49.199 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:49.199 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:49.199 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:49.199 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:49.200 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:49.200 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:49.200 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:49.200 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:49.200 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:49.200 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:49.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:49.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:49.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:49.304 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:49.304 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:49.304 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:49.304 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:49.304 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:49.305 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:49.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:49.680 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:49.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:49.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:50.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:50.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:50.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:50.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:50.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:50.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:50.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:50.238 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:50.238 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:50.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599494 2023-11-02T09:35:50.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599494 '!=' null 2023-11-02T09:35:50.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:50.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:50.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:50.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:51.058 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:51.058 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:51.058 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:51.058 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:51.059 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:51.059 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:51.059 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:51.059 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:51.059 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:51.060 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:51.060 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:51.060 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:51.060 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:51.060 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:51.060 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:51.061 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:51.061 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:51.061 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:51.061 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:51.061 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:51.062 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:51.062 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:51.062 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:51.062 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:51.062 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:51.063 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:51.063 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:51.063 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:51.063 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:51.063 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:51.064 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:51.064 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:51.064 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:51.064 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:51.064 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:51.065 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:51.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:51.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:51.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:51.167 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:51.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:51.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:51.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:51.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:51.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:51.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:51.532 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:51.533 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:51.533 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:52.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:52.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:52.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:52.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:52.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:52.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:52.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:52.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:52.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:52.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209512 2023-11-02T09:35:52.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209512 '!=' null 2023-11-02T09:35:52.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:52.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:52.798 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:52.798 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:52.875 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:52.875 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:52.875 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:52.875 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:52.876 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:52.876 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:52.876 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:52.876 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:52.877 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:52.877 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:52.877 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:52.877 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:52.877 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:52.878 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:52.878 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:52.878 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:52.878 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:52.878 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:52.879 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:52.879 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:52.879 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:52.879 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:52.879 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:52.879 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:52.880 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:52.880 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:52.880 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:52.880 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:52.881 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:52.881 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:52.881 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:52.881 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:52.881 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:52.882 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:52.882 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:52.882 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:52.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:52.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:52.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:52.984 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:52.984 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:52.984 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:52.984 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:52.985 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:52.985 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:53.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:53.350 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:53.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:53.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:53.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:53.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:53.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:53.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:53.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:53.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:53.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:53.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:53.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:54.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209512 2023-11-02T09:35:54.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209512 '!=' null 2023-11-02T09:35:54.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:54.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:54.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:54.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:54.569 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:54.569 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:54.569 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:54.569 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:54.570 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:54.570 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:54.570 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:54.570 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:54.570 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:54.571 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:54.571 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:54.571 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:54.571 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:54.571 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:54.572 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:54.572 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:54.572 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:54.572 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:54.572 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:54.573 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:54.573 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:54.573 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:54.573 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:54.573 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:54.573 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:54.574 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:54.575 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:54.575 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:54.575 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:54.575 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:54.576 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:54.576 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:54.576 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:54.576 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:54.576 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:54.577 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:54.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:54.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:54.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:54.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:54.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:54.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:54.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:54.682 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:54.682 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:55.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:55.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:55.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:55.050 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:55.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:55.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:55.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:55.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:55.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:55.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:55.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:55.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:55.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:56.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18346560 2023-11-02T09:35:56.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18346560 '!=' null 2023-11-02T09:35:56.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:56.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:56.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:56.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:56.294 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:56.294 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:56.294 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:56.294 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:56.295 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:56.295 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:56.295 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:56.295 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:56.295 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:56.296 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:56.296 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:56.296 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:56.296 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:56.296 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:56.297 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:56.297 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:56.297 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:56.297 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:56.297 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:56.297 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:56.298 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:56.298 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:56.298 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:56.298 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:56.298 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:56.299 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:56.299 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:56.299 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:56.300 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:56.300 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:56.300 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:56.300 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:56.300 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:56.301 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:56.301 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:56.301 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:56.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:56.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:56.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:56.403 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:56.403 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:56.403 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:56.403 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:56.404 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:56.404 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:56.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:56.859 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:56.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:56.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:57.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:57.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:57.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:57.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:57.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:57.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:57.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:57.399 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:57.399 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:57.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18346524 2023-11-02T09:35:57.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18346524 '!=' null 2023-11-02T09:35:57.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:57.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:57.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:57.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:58.011 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:58.012 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:58.012 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:58.012 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:58.013 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:58.013 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:58.013 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:58.013 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:58.013 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:58.014 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:58.014 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:58.014 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:58.014 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:58.015 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:58.015 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:58.015 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:58.015 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:58.016 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:58.016 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:58.016 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:58.016 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:58.016 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:58.017 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:58.017 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:58.017 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:58.018 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:58.018 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:58.018 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:58.018 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:58.019 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:58.021 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:58.021 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:58.021 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:58.022 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:58.022 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:58.022 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:58.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:58.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:58.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:58.121 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:58.121 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:58.121 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:58.121 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:58.122 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:58.122 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:35:58.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:35:58.489 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:35:58.490 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:35:58.490 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:35:59.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:35:59.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:35:59.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:35:59.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:35:59.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:35:59.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:35:59.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:35:59.089 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:35:59.089 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:35:59.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27083129 2023-11-02T09:35:59.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27083129 '!=' null 2023-11-02T09:35:59.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:35:59.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:59.602 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:35:59.602 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:35:59.681 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:35:59.681 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:35:59.681 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:59.681 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:59.682 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:35:59.682 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:59.682 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:35:59.682 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:35:59.682 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:59.683 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:35:59.683 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:59.683 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:59.683 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:59.683 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:59.683 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:35:59.684 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:35:59.684 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:59.684 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:59.684 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:59.684 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:59.685 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:35:59.685 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:35:59.685 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:35:59.685 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:35:59.685 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:35:59.685 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:35:59.686 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:59.686 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:35:59.686 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:35:59.686 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:35:59.686 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:35:59.687 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:35:59.690 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:35:59.691 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:35:59.691 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:35:59.691 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:35:59.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:35:59.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:35:59.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:35:59.790 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:35:59.790 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:35:59.790 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:35:59.790 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:35:59.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:35:59.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:00.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:00.171 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:00.171 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:00.171 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:00.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:00.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:00.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:00.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:00.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:00.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:00.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:00.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:00.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:01.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18346653 2023-11-02T09:36:01.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18346653 '!=' null 2023-11-02T09:36:01.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:01.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:01.330 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:01.330 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:01.406 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:01.407 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:01.407 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:01.407 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:01.407 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:01.407 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:01.408 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:01.408 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:01.408 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:01.408 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:01.408 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:01.408 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:01.409 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:01.409 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:01.409 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:01.409 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:01.409 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:01.410 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:01.410 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:01.410 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:01.410 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:01.410 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:01.411 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:01.411 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:01.411 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:01.411 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:01.411 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:01.412 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:01.412 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:01.412 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:01.412 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:01.412 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:01.413 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:01.413 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:01.413 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:01.413 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:01.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:01.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:01.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:01.515 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:01.515 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:01.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:01.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:01.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:01.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:01.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:01.885 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:01.885 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:01.886 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:02.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:02.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:02.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:02.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:02.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:02.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:02.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:02.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:02.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:03.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209720 2023-11-02T09:36:03.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209720 '!=' null 2023-11-02T09:36:03.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:03.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:03.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:03.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:03.179 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:03.180 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:03.180 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:03.180 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:03.180 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:03.181 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:03.181 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:03.181 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:03.181 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:03.181 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:03.181 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:03.182 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:03.182 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:03.182 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:03.182 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:03.182 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:03.183 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:03.183 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:03.183 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:03.183 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:03.184 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:03.184 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:03.184 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:03.184 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:03.185 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:03.185 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:03.185 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:03.185 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:03.186 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:03.186 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:03.186 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:03.186 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:03.186 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:03.186 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:03.187 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:03.187 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:03.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:03.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:03.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:03.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:03.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:03.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:03.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:03.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:03.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:03.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:03.655 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:03.656 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:03.656 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:04.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:04.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:04.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:04.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:04.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:04.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:04.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:04.162 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:04.162 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:04.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599629 2023-11-02T09:36:04.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599629 '!=' null 2023-11-02T09:36:04.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:04.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:04.869 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:04.869 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:04.945 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:04.946 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:04.946 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:04.946 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:04.946 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:04.947 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:04.947 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:04.947 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:04.947 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:04.947 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:04.948 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:04.948 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:04.948 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:04.948 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:04.948 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:04.949 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:04.949 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:04.949 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:04.949 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:04.949 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:04.950 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:04.950 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:04.950 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:04.950 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:04.950 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:04.951 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:04.951 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:04.951 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:04.951 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:04.952 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:04.952 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:04.952 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:04.952 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:04.952 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:04.952 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:04.953 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:04.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:05.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:05.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:05.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:05.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:05.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:05.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:05.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:05.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:05.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:05.425 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:05.426 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:05.426 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:06.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:06.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:06.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:06.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:06.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:06.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:06.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:06.161 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:06.161 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:06.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599656 2023-11-02T09:36:06.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599656 '!=' null 2023-11-02T09:36:06.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:06.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:06.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:06.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:06.778 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:06.778 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:06.778 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:06.778 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:06.779 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:06.779 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:06.779 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:06.779 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:06.779 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:06.780 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:06.780 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:06.780 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:06.780 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:06.780 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:06.781 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:06.781 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:06.781 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:06.781 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:06.781 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:06.781 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:06.782 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:06.782 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:06.782 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:06.782 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:06.782 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:06.783 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:06.783 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:06.783 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:06.783 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:06.783 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:06.783 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:06.784 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:06.784 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:06.784 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:06.784 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:06.784 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:06.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:06.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:06.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:06.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:06.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:06.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:06.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:06.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:06.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:07.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:07.258 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:07.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:07.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:07.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:07.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:07.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:07.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:07.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:07.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:07.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:07.832 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:07.833 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:08.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599656 2023-11-02T09:36:08.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599656 '!=' null 2023-11-02T09:36:08.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:08.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:08.409 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:08.409 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:08.486 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:08.486 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:08.486 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:08.486 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:08.487 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:08.487 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:08.487 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:08.487 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:08.488 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:08.488 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:08.488 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:08.488 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:08.488 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:08.489 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:08.489 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:08.489 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:08.489 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:08.489 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:08.490 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:08.490 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:08.490 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:08.490 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:08.490 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:08.491 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:08.491 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:08.491 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:08.491 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:08.491 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:08.492 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:08.492 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:08.492 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:08.492 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:08.492 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:08.493 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:08.493 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:08.493 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:08.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:08.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:08.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:08.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:08.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:08.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:08.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:08.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:08.597 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:09.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:09.084 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:09.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:09.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:09.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:09.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:09.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:09.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:09.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:09.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:09.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:09.588 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:09.588 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:10.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210072 2023-11-02T09:36:10.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210072 '!=' null 2023-11-02T09:36:10.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:10.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:10.157 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:10.157 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:10.231 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:10.231 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:10.232 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:10.232 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:10.232 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:10.233 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:10.233 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:10.234 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:10.234 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:10.234 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:10.234 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:10.235 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:10.235 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:10.235 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:10.236 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:10.236 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:10.237 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:10.237 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:10.237 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:10.238 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:10.238 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:10.238 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:10.239 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:10.239 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:10.239 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:10.240 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:10.240 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:10.240 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:10.241 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:10.241 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:10.241 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:10.242 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:10.242 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:10.242 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:10.242 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:10.243 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:10.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:10.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:10.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:10.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:10.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:10.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:10.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:10.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:10.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:10.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:10.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:10.783 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:10.783 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:11.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:11.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:11.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:11.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:11.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:11.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:11.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:11.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:11.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:11.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473503 2023-11-02T09:36:11.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473503 '!=' null 2023-11-02T09:36:11.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:11.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:11.845 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:11.845 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:11.921 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:11.921 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:11.921 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:11.922 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:11.922 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:11.922 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:11.922 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:11.922 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:11.923 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:11.923 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:11.923 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:11.924 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:11.924 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:11.924 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:11.925 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:11.925 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:11.925 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:11.925 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:11.925 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:11.926 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:11.926 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:11.926 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:11.926 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:11.926 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:11.926 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:11.927 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:11.927 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:11.927 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:11.927 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:11.928 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:11.928 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:11.928 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:11.929 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:11.929 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:11.929 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:11.929 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:11.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:12.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:12.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:12.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:12.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:12.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:12.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:12.032 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:12.032 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:12.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:12.398 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:12.398 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:12.399 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:13.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:13.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:13.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:13.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:13.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:13.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:13.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:13.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:13.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:13.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210212 2023-11-02T09:36:13.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210212 '!=' null 2023-11-02T09:36:13.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:13.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:13.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:13.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:13.698 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:13.698 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:13.699 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:13.699 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:13.699 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:13.699 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:13.699 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:13.700 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:13.700 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:13.700 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:13.700 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:13.700 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:13.701 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:13.701 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:13.701 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:13.701 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:13.702 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:13.702 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:13.702 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:13.702 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:13.702 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:13.703 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:13.703 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:13.703 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:13.703 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:13.703 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:13.704 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:13.704 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:13.704 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:13.704 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:13.704 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:13.705 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:13.705 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:13.705 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:13.705 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:13.706 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:13.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:13.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:13.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:13.807 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:13.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:13.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:13.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:13.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:13.809 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:14.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:14.176 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:14.176 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:14.176 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:14.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:14.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:14.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:14.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:14.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:14.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:14.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:14.685 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:14.686 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:15.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18347107 2023-11-02T09:36:15.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18347107 '!=' null 2023-11-02T09:36:15.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:15.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:15.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:15.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:15.324 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:15.326 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:15.326 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:15.326 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:15.326 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:15.326 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:15.327 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:15.327 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:15.327 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:15.327 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:15.327 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:15.328 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:15.328 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:15.328 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:15.328 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:15.328 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:15.329 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:15.329 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:15.329 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:15.329 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:15.329 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:15.329 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:15.330 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:15.330 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:15.330 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:15.330 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:15.330 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:15.331 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:15.331 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:15.331 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:15.331 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:15.331 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:15.332 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:15.332 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:15.332 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:15.333 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:15.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:15.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:15.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:15.434 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:15.434 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:15.435 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:15.435 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:15.435 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:15.435 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:15.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:15.807 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:15.807 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:15.807 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:16.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:16.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:16.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:16.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:16.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:16.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:16.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:16.398 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:16.398 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:16.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18347163 2023-11-02T09:36:16.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18347163 '!=' null 2023-11-02T09:36:16.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:16.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:16.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:16.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:17.012 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:17.013 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:17.013 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:17.013 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:17.013 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:17.014 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:17.014 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:17.014 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:17.014 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:17.015 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:17.015 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:17.015 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:17.015 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:17.016 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:17.016 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:17.016 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:17.016 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:17.016 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:17.017 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:17.017 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:17.017 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:17.018 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:17.018 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:17.018 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:17.018 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:17.018 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:17.019 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:17.019 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:17.019 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:17.019 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:17.019 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:17.019 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:17.020 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:17.020 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:17.020 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:17.021 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:17.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:17.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:17.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:17.122 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:17.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:17.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:17.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:17.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:17.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:17.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:17.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:17.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:17.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:18.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:18.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:18.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:18.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:18.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:18.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:18.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:18.157 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:18.157 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:18.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27083799 2023-11-02T09:36:18.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27083799 '!=' null 2023-11-02T09:36:18.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:18.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:18.833 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:18.834 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:18.910 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:18.911 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:18.911 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:18.911 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:18.911 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:18.911 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:18.912 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:18.912 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:18.912 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:18.912 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:18.912 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:18.912 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:18.913 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:18.913 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:18.913 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:18.913 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:18.913 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:18.914 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:18.914 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:18.914 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:18.914 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:18.914 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:18.915 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:18.915 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:18.916 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:18.916 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:18.916 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:18.917 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:18.917 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:18.918 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:18.918 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:18.918 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:18.918 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:18.919 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:18.919 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:18.919 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:18.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:19.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:19.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:19.019 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:19.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:19.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:19.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:19.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:19.021 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:19.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:19.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:19.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:19.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:19.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:19.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:19.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:19.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:19.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:19.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:19.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:19.974 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:19.974 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:20.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473407 2023-11-02T09:36:20.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473407 '!=' null 2023-11-02T09:36:20.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:20.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:20.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:20.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:20.836 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:20.836 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:20.836 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:20.837 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:20.837 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:20.837 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:20.837 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:20.838 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:20.838 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:20.838 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:20.838 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:20.838 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:20.839 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:20.839 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:20.839 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:20.839 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:20.839 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:20.840 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:20.840 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:20.840 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:20.840 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:20.840 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:20.841 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:20.841 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:20.841 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:20.841 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:20.841 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:20.842 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:20.842 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:20.842 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:20.842 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:20.843 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:20.843 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:20.843 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:20.843 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:20.843 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:20.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:20.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:20.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:20.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:20.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:20.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:20.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:20.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:20.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:21.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:21.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:21.313 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:21.313 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:21.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:21.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:21.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:21.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:21.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:21.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:21.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:21.827 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:21.827 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:22.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473407 2023-11-02T09:36:22.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473407 '!=' null 2023-11-02T09:36:22.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:22.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:22.456 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:22.456 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:22.532 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:22.533 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:22.533 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:22.533 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:22.533 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:22.534 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:22.534 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:22.534 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:22.534 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:22.534 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:22.535 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:22.535 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:22.535 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:22.535 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:22.536 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:22.536 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:22.536 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:22.536 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:22.536 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:22.537 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:22.537 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:22.537 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:22.538 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:22.538 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:22.538 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:22.538 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:22.539 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:22.539 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:22.540 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:22.540 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:22.540 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:22.540 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:22.540 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:22.541 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:22.541 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:22.541 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:22.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:22.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:22.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:22.642 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:22.642 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:22.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:22.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:22.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:22.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:23.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:23.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:23.058 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:23.058 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:23.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:23.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:23.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:23.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:23.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:23.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:23.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:23.561 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:23.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:24.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209975 2023-11-02T09:36:24.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209975 '!=' null 2023-11-02T09:36:24.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:24.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:24.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:24.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:24.356 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:24.356 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:24.356 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:24.356 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:24.357 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:24.357 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:24.357 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:24.357 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:24.357 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:24.358 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:24.358 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:24.358 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:24.358 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:24.358 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:24.359 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:24.359 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:24.359 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:24.359 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:24.359 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:24.360 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:24.360 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:24.360 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:24.360 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:24.360 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:24.361 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:24.361 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:24.361 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:24.361 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:24.361 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:24.362 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:24.362 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:24.362 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:24.362 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:24.362 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:24.363 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:24.363 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:24.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:24.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:24.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:24.465 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:24.465 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:24.466 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:24.466 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:24.466 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:24.466 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:24.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:24.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:24.938 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:24.938 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:25.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:25.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:25.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:25.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:25.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:25.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:25.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:25.447 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:25.448 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:26.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472987 2023-11-02T09:36:26.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472987 '!=' null 2023-11-02T09:36:26.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:26.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:26.071 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:26.071 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:26.146 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:26.147 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:26.147 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:26.147 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:26.147 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:26.148 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:26.148 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:26.148 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:26.148 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:26.148 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:26.149 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:26.149 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:26.149 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:26.149 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:26.149 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:26.150 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:26.150 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:26.150 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:26.150 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:26.150 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:26.150 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:26.151 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:26.151 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:26.151 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:26.151 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:26.151 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:26.152 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:26.152 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:26.152 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:26.152 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:26.152 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:26.153 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:26.153 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:26.153 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:26.153 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:26.153 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:26.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:26.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:26.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:26.256 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:26.256 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:26.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:26.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:26.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:26.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:26.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:26.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:26.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:26.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:27.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:27.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:27.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:27.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:27.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:27.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:27.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:27.139 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:27.139 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:27.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472966 2023-11-02T09:36:27.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472966 '!=' null 2023-11-02T09:36:27.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:27.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:27.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:27.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:27.871 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:27.871 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:27.871 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:27.872 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:27.872 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:27.872 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:27.872 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:27.872 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:27.873 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:27.873 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:27.873 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:27.873 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:27.873 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:27.873 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:27.874 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:27.874 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:27.874 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:27.874 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:27.874 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:27.875 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:27.875 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:27.875 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:27.875 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:27.875 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:27.876 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:27.876 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:27.876 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:27.876 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:27.876 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:27.877 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:27.877 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:27.877 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:27.877 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:27.877 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:27.877 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:27.878 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:27.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:27.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:27.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:27.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:27.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:27.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:27.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:27.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:27.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:28.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:28.344 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:28.344 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:28.345 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:28.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:28.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:28.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:28.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:28.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:28.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:28.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:28.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:28.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:29.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209385 2023-11-02T09:36:29.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209385 '!=' null 2023-11-02T09:36:29.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:29.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:29.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:29.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:29.457 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:29.458 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:29.458 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:29.458 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:29.458 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:29.458 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:29.459 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:29.459 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:29.459 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:29.459 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:29.459 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:29.460 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:29.460 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:29.460 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:29.460 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:29.460 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:29.461 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:29.461 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:29.461 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:29.461 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:29.461 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:29.462 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:29.462 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:29.462 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:29.462 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:29.462 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:29.463 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:29.464 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:29.465 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:29.465 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:29.465 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:29.465 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:29.466 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:29.466 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:29.466 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:29.466 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:29.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:29.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:29.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:29.566 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:29.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:29.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:29.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:29.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:29.568 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:29.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:29.937 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:29.937 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:29.937 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:30.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:30.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:30.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:30.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:30.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:30.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:30.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:30.445 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:30.445 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:31.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472936 2023-11-02T09:36:31.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472936 '!=' null 2023-11-02T09:36:31.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:31.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:31.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:31.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:31.200 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:31.200 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:31.200 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:31.200 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:31.201 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:31.201 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:31.201 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:31.201 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:31.201 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:31.202 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:31.202 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:31.202 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:31.202 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:31.202 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:31.203 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:31.203 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:31.203 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:31.203 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:31.203 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:31.204 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:31.204 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:31.204 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:31.204 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:31.204 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:31.205 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:31.205 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:31.205 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:31.205 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:31.206 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:31.206 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:31.206 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:31.206 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:31.206 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:31.207 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:31.207 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:31.207 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:31.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:31.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:31.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:31.309 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:31.309 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:31.309 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:31.310 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:31.310 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:31.310 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:31.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:31.720 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:31.721 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:31.721 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:32.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:32.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:32.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:32.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:32.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:32.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:32.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:32.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:32.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:32.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209424 2023-11-02T09:36:32.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209424 '!=' null 2023-11-02T09:36:32.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:32.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:32.906 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:32.906 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:32.983 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:32.983 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:32.983 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:32.983 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:32.984 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:32.984 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:32.984 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:32.984 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:32.984 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:32.985 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:32.985 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:32.985 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:32.985 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:32.985 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:32.986 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:32.986 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:32.986 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:32.986 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:32.986 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:32.987 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:32.987 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:32.987 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:32.987 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:32.987 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:32.988 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:32.988 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:32.988 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:32.988 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:32.988 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:32.989 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:32.989 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:32.989 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:32.989 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:32.989 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:32.990 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:32.990 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:32.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:33.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:33.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:33.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:33.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:33.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:33.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:33.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:33.097 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:33.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:33.471 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:33.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:33.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:34.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:34.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:34.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:34.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:34.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:34.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:34.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:34.028 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:34.028 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:34.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209424 2023-11-02T09:36:34.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209424 '!=' null 2023-11-02T09:36:34.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:34.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:34.602 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:34.603 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:34.679 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:34.679 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:34.679 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:34.680 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:34.680 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:34.680 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:34.680 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:34.680 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:34.681 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:34.681 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:34.681 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:34.681 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:34.681 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:34.682 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:34.682 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:34.682 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:34.682 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:34.682 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:34.683 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:34.683 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:34.683 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:34.683 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:34.683 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:34.684 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:34.684 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:34.684 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:34.684 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:34.685 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:34.685 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:34.685 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:34.685 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:34.685 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:34.685 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:34.686 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:34.686 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:34.686 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:34.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:34.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:34.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:34.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:34.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:34.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:34.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:34.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:34.790 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:35.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:35.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:35.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:35.166 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:35.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:35.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:35.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:35.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:35.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:35.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:35.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:35.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:35.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:36.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472789 2023-11-02T09:36:36.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472789 '!=' null 2023-11-02T09:36:36.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:36.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:36.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:36.258 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:36.334 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:36.334 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:36.335 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:36.335 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:36.335 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:36.335 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:36.335 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:36.336 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:36.336 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:36.336 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:36.336 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:36.336 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:36.337 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:36.337 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:36.337 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:36.337 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:36.337 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:36.337 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:36.338 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:36.338 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:36.338 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:36.339 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:36.339 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:36.340 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:36.340 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:36.340 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:36.340 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:36.340 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:36.341 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:36.341 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:36.341 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:36.341 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:36.341 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:36.342 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:36.342 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:36.342 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:36.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:36.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:36.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:36.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:36.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:36.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:36.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:36.445 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:36.445 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:36.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:36.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:36.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:36.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:37.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:37.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:37.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:37.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:37.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:37.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:37.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:37.371 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:37.371 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:37.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473064 2023-11-02T09:36:37.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473064 '!=' null 2023-11-02T09:36:37.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:37.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:37.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:37.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:38.022 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:38.023 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:38.023 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:38.023 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:38.023 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:38.024 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:38.024 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:38.024 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:38.024 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:38.025 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:38.025 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:38.025 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:38.025 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:38.025 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:38.026 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:38.026 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:38.026 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:38.028 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:38.028 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:38.028 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:38.028 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:38.028 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:38.029 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:38.029 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:38.029 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:38.029 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:38.030 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:38.030 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:38.030 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:38.030 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:38.030 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:38.031 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:38.031 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:38.031 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:38.031 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:38.031 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:38.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:38.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:38.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:38.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:38.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:38.133 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:38.133 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:38.133 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:38.133 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:38.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:38.583 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:38.584 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:38.584 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:39.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:39.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:39.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:39.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:39.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:39.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:39.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:39.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:39.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:39.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25335572 2023-11-02T09:36:39.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25335572 '!=' null 2023-11-02T09:36:39.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:39.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:39.762 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:39.762 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:39.839 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:39.839 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:39.840 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:39.840 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:39.840 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:39.840 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:39.841 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:39.841 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:39.841 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:39.841 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:39.841 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:39.841 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:39.842 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:39.842 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:39.842 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:39.842 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:39.842 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:39.843 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:39.843 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:39.843 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:39.845 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:39.845 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:39.845 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:39.846 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:39.846 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:39.846 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:39.846 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:39.846 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:39.847 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:39.847 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:39.847 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:39.847 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:39.848 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:39.848 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:39.848 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:39.848 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:39.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:39.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:39.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:39.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:39.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:39.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:39.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:39.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:39.950 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:40.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:40.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:40.343 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:40.343 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:40.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:40.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:40.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:40.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:40.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:40.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:40.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:40.918 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:40.918 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:41.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599172 2023-11-02T09:36:41.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599172 '!=' null 2023-11-02T09:36:41.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:41.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:41.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:41.509 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:41.584 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:41.585 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:41.585 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:41.585 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:41.585 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:41.585 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:41.586 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:41.586 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:41.586 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:41.586 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:41.586 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:41.587 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:41.587 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:41.587 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:41.587 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:41.587 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:41.588 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:41.588 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:41.588 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:41.588 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:41.588 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:41.588 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:41.589 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:41.589 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:41.589 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:41.589 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:41.589 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:41.590 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:41.590 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:41.590 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:41.590 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:41.590 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:41.590 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:41.591 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:41.591 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:41.591 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:41.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:41.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:41.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:41.694 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:41.694 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:41.694 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:41.695 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:41.695 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:41.695 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:42.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:42.065 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:42.066 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:42.066 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:42.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:42.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:42.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:42.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:42.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:42.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:42.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:42.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:42.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:43.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25335493 2023-11-02T09:36:43.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25335493 '!=' null 2023-11-02T09:36:43.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:43.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:43.184 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:43.184 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:43.261 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:43.261 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:43.261 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:43.261 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:43.262 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:43.262 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:43.262 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:43.262 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:43.262 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:43.263 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:43.263 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:43.263 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:43.263 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:43.263 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:43.263 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:43.264 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:43.264 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:43.264 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:43.264 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:43.264 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:43.265 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:43.265 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:43.265 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:43.265 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:43.265 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:43.266 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:43.266 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:43.267 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:43.267 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:43.267 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:43.267 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:43.269 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:43.269 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:43.269 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:43.270 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:43.270 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:43.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:43.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:43.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:43.370 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:43.370 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:43.371 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:43.371 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:43.371 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:43.371 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:43.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:43.764 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:43.764 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:43.764 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:44.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:44.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:44.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:44.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:44.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:44.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:44.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:44.270 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:44.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:44.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599112 2023-11-02T09:36:44.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599112 '!=' null 2023-11-02T09:36:44.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:44.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:44.926 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:44.926 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:45.003 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:45.003 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:45.004 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:45.004 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:45.004 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:45.004 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:45.004 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:45.005 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:45.005 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:45.005 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:45.006 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:45.006 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:45.006 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:45.006 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:45.007 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:45.007 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:45.007 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:45.007 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:45.008 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:45.008 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:45.008 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:45.009 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:45.009 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:45.009 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:45.009 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:45.009 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:45.010 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:45.010 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:45.010 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:45.010 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:45.010 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:45.011 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:45.011 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:45.011 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:45.011 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:45.011 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:45.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:45.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:45.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:45.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:45.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:45.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:45.116 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:45.116 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:45.116 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:45.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:45.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:45.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:45.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:46.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:46.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:46.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:46.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:46.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:46.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:46.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:46.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:46.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:46.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599345 2023-11-02T09:36:46.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599345 '!=' null 2023-11-02T09:36:46.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:46.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:46.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:46.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:46.754 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:46.755 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:46.755 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:46.755 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:46.755 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:46.755 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:46.756 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:46.756 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:46.756 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:46.756 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:46.756 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:46.756 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:46.757 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:46.757 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:46.757 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:46.757 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:46.757 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:46.758 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:46.758 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:46.758 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:46.758 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:46.758 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:46.759 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:46.759 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:46.759 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:46.759 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:46.759 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:46.759 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:46.760 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:46.760 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:46.760 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:46.760 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:46.760 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:46.761 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:46.761 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:46.762 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:46.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:46.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:46.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:46.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:46.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:46.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:46.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:46.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:46.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:47.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:47.279 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:47.279 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:47.279 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:47.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:47.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:47.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:47.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:47.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:47.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:47.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:47.774 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:47.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:48.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599345 2023-11-02T09:36:48.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599345 '!=' null 2023-11-02T09:36:48.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:48.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:48.318 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:48.318 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:48.395 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:48.396 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:48.396 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:48.396 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:48.396 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:48.397 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:48.397 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:48.397 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:48.397 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:48.397 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:48.398 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:48.398 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:48.398 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:48.398 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:48.399 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:48.399 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:48.399 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:48.399 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:48.400 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:48.400 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:48.400 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:48.400 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:48.400 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:48.401 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:48.401 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:48.401 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:48.401 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:48.402 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:48.402 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:48.402 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:48.402 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:48.402 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:48.403 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:48.403 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:48.403 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:48.403 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:48.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:48.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:48.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:48.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:48.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:48.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:48.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:48.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:48.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:48.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:48.972 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:48.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:48.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:49.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:49.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:49.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:49.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:49.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:49.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:49.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:49.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:49.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:50.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25335688 2023-11-02T09:36:50.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25335688 '!=' null 2023-11-02T09:36:50.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:50.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:50.100 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:50.100 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:50.177 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:50.178 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:50.178 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:50.178 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:50.178 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:50.178 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:50.179 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:50.179 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:50.179 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:50.179 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:50.179 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:50.180 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:50.180 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:50.180 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:50.180 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:50.180 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:50.180 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:50.181 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:50.181 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:50.181 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:50.181 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:50.181 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:50.182 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:50.182 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:50.182 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:50.182 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:50.182 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:50.183 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:50.183 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:50.183 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:50.183 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:50.183 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:50.183 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:50.184 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:50.184 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:50.184 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:50.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:50.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:50.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:50.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:50.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:50.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:50.288 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:50.288 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:50.288 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:50.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:50.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:50.675 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:50.675 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:51.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:51.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:51.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:51.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:51.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:51.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:51.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:51.393 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:51.393 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:51.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473195 2023-11-02T09:36:51.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473195 '!=' null 2023-11-02T09:36:51.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:51.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:51.922 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:51.922 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:51.999 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:52.000 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:52.000 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:52.000 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:52.000 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:52.000 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:52.001 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:52.001 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:52.001 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:52.001 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:52.001 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:52.002 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:52.002 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:52.002 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:52.002 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:52.002 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:52.003 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:52.003 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:52.003 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:52.003 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:52.003 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:52.004 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:52.004 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:52.004 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:52.004 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:52.004 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:52.005 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:52.005 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:52.005 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:52.005 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:52.005 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:52.005 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:52.006 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:52.006 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:52.006 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:52.006 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:52.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:52.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:52.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:52.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:52.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:52.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:52.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:52.110 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:52.110 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:52.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:52.475 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:52.475 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:52.476 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:52.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:52.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:52.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:52.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:52.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:52.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:52.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:52.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:52.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:53.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209807 2023-11-02T09:36:53.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209807 '!=' null 2023-11-02T09:36:53.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:53.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:53.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:53.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:53.677 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:53.677 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:53.677 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:53.677 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:53.678 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:53.678 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:53.678 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:53.678 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:53.678 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:53.679 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:53.679 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:53.679 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:53.679 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:53.679 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:53.680 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:53.680 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:53.680 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:53.680 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:53.680 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:53.681 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:53.681 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:53.681 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:53.681 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:53.681 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:53.681 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:53.682 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:53.682 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:53.682 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:53.682 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:53.682 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:53.683 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:53.683 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:53.683 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:53.683 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:53.683 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:53.683 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:53.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:53.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:53.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:53.786 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:53.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:53.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:53.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:53.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:53.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:54.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:54.155 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:54.155 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:54.155 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:54.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:54.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:54.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:54.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:54.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:54.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:54.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:54.898 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:54.898 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:55.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473374 2023-11-02T09:36:55.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473374 '!=' null 2023-11-02T09:36:55.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:55.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:55.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:55.429 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:55.504 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:55.505 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:55.505 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:55.505 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:55.505 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:55.506 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:55.506 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:55.506 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:55.506 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:55.506 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:55.506 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:55.507 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:55.507 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:55.507 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:55.507 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:55.507 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:55.508 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:55.508 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:55.508 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:55.508 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:55.508 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:55.509 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:55.509 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:55.509 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:55.509 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:55.509 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:55.509 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:55.510 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:55.510 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:55.510 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:55.510 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:55.510 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:55.511 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:55.511 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:55.511 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:55.511 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:55.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:55.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:55.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:55.614 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:55.614 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:55.614 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:55.614 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:55.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:55.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:55.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:55.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:55.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:55.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:56.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:56.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:56.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:56.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:56.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:56.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:56.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:56.660 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:56.660 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:57.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473366 2023-11-02T09:36:57.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473366 '!=' null 2023-11-02T09:36:57.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:57.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:57.311 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:57.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:57.390 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:57.390 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:57.391 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:57.391 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:57.391 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:57.391 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:57.391 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:57.392 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:57.392 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:57.392 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:57.392 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:57.393 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:57.393 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:57.393 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:57.393 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:57.393 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:57.394 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:57.394 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:57.394 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:57.394 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:57.394 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:57.395 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:57.395 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:57.395 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:57.395 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:57.395 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:57.396 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:57.396 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:57.396 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:57.396 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:57.396 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:57.397 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:57.397 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:57.397 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:57.397 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:57.397 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:57.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:57.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:57.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:57.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:57.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:57.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:57.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:57.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:57.498 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:57.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:57.861 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:57.861 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:57.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:36:58.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:36:58.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:36:58.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:36:58.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:36:58.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:36:58.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:36:58.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:36:58.476 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:36:58.477 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:36:58.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336247 2023-11-02T09:36:58.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336247 '!=' null 2023-11-02T09:36:58.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:36:58.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:58.983 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:36:58.984 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:36:59.060 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:36:59.060 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:36:59.061 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:59.061 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:59.061 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:36:59.062 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:59.062 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:36:59.062 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:36:59.063 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:59.063 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:36:59.063 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:59.064 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:59.064 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:59.064 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:59.065 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:36:59.065 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:36:59.065 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:59.066 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:59.066 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:59.067 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:59.067 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:36:59.067 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:36:59.068 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:36:59.068 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "7.0", 2023-11-02T09:36:59.068 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 255, 2023-11-02T09:36:59.069 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:36:59.069 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:59.069 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:36:59.070 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:36:59.070 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:36:59.070 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:36:59.071 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:36:59.071 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:36:59.072 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:36:59.072 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:36:59.072 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:36:59.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:36:59.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:36:59.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:36:59.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:36:59.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:36:59.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:36:59.169 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:36:59.169 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:36:59.169 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:36:59.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:36:59.540 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:36:59.540 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:36:59.541 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:00.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:37:00.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:37:00.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:00.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:00.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:00.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:00.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:00.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:00.154 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:00.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336247 2023-11-02T09:37:00.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336247 '!=' null 2023-11-02T09:37:00.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:00.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:00.801 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:00.801 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:00.877 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:00.877 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:00.878 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:00.878 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:00.878 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:00.879 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:00.879 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:00.879 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:00.880 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:00.880 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:00.880 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:00.881 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:00.881 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:00.882 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:00.882 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:00.883 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:00.883 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:00.883 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:00.883 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:00.884 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:00.884 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:00.884 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:00.885 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:00.885 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:00.885 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:00.886 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:00.886 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:00.886 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:00.887 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:00.887 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:00.888 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:00.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:00.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:00.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:00.986 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:00.987 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:00.987 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:00.987 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:00.987 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:00.987 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:01.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:37:01.592 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:01.592 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:01.593 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:02.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:37:02.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:37:02.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:02.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:02.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:02.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:02.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:02.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:02.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:02.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588844 2023-11-02T09:37:02.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588844 '!=' null 2023-11-02T09:37:02.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:02.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:02.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:02.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:02.940 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:02.940 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:02.941 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:02.941 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:02.941 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:02.941 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:02.941 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:02.942 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:02.942 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:02.942 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:02.942 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:02.942 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:02.943 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:02.943 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:02.943 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:02.943 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:02.943 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:02.943 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:02.944 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:02.944 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:02.944 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:02.944 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:02.944 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:02.945 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:02.945 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:02.945 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:02.945 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:02.945 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:02.945 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:02.946 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:02.946 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:02.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:03.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:03.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:03.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:03.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:03.050 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:03.050 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:03.050 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:03.050 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:03.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:37:03.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:03.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:03.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:03.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:37:03.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:37:03.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:03.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:03.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:03.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:03.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:03.986 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:03.986 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:04.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588844 2023-11-02T09:37:04.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588844 '!=' null 2023-11-02T09:37:04.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:04.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:04.617 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:04.617 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:04.693 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:04.694 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:04.694 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:04.694 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:04.694 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:04.695 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:04.695 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:04.695 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:04.695 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:04.695 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:04.696 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:04.696 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:04.696 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:04.696 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:04.696 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:04.697 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:04.697 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:04.697 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:04.697 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:04.697 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:04.697 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:04.698 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:04.698 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:04.698 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:04.698 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:04.698 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:04.699 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:04.699 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:04.699 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:04.699 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:04.699 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:04.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:04.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:04.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:04.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:04.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:04.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:04.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:04.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:04.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:05.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:37:05.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:05.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:05.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:05.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:37:05.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:37:05.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:05.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:05.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:05.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:05.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:05.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:05.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:06.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588753 2023-11-02T09:37:06.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588753 '!=' null 2023-11-02T09:37:06.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:06.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:06.308 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:06.308 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:06.385 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:06.386 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:06.386 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:06.386 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:06.386 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:06.386 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:06.387 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:06.387 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:06.387 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:06.387 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:06.387 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:06.388 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:06.388 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:06.388 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:06.388 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:06.388 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:06.389 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:06.389 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:06.389 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:06.389 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:06.389 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:06.390 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:06.390 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:06.390 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:06.390 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:06.390 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:06.391 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:06.391 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:06.391 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:06.391 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:06.391 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:06.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:06.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:06.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:06.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:06.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:06.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:06.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:06.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:06.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:06.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-11-02T09:37:06.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:06.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:06.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:07.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2023-11-02T09:37:07.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-11-02T09:37:07.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:07.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:07.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:07.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:07.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:07.369 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:07.369 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:08.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11794433 2023-11-02T09:37:08.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11794433 '!=' null 2023-11-02T09:37:08.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:08.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:08.026 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:08.026 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:08.102 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:08.103 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:08.103 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:08.103 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:08.103 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:08.104 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:08.104 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:08.104 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:08.104 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:08.104 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:08.104 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:08.105 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:08.105 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:08.105 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:08.105 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:08.105 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:08.106 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:08.106 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:08.106 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:08.106 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:08.106 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:08.107 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:08.107 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:08.107 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:08.107 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:08.107 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:08.108 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:08.108 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:08.108 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:08.108 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:08.108 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:08.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:08.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:08.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:08.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:08.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:08.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:08.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:08.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:08.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:08.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:08.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:08.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:08.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:09.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:09.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 0 2023-11-02T09:37:09.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2023-11-02T09:37:09.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2023-11-02T09:37:09.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:09.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:09.183 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:09.256 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:09.256 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:09.256 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:09.257 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:09.257 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:09.257 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:09.257 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:09.257 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:09.258 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:09.258 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:09.258 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:09.258 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:09.258 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:09.259 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:09.259 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:09.259 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:09.259 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:09.259 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:09.259 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:09.260 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:09.260 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:09.260 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:09.260 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:09.260 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:09.261 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:09.261 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:09.262 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:09.263 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:09.263 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:09.263 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:09.263 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:09.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:09.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:09.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:09.365 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:09.366 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:09.366 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:09.366 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:09.366 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:09.367 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:09.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:09.739 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:09.739 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:09.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:10.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:10.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:10.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:10.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:10.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:10.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:10.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:10.244 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:10.244 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:10.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19919474 2023-11-02T09:37:10.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19919474 '!=' null 2023-11-02T09:37:10.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:10.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:10.855 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:10.856 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:10.932 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:10.932 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:10.932 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:10.933 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:10.933 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:10.933 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:10.933 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:10.933 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:10.934 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:10.934 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:10.934 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:10.934 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:10.934 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:10.935 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:10.935 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:10.935 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:10.935 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:10.935 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:10.936 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:10.936 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:10.936 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:10.936 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:10.936 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:10.937 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:10.937 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:10.937 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:10.937 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:10.937 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:10.938 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:10.938 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:10.938 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:10.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:11.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:11.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:11.042 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:11.042 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:11.042 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:11.042 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:11.042 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:11.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:11.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:11.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:11.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:11.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:12.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:12.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:12.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:12.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:12.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:12.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:12.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:12.035 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:12.035 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:12.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27390399 2023-11-02T09:37:12.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27390399 '!=' null 2023-11-02T09:37:12.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:12.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:12.583 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:12.583 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:12.660 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:12.661 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:12.661 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:12.661 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:12.661 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:12.661 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:12.662 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:12.662 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:12.662 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:12.662 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:12.662 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:12.663 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:12.663 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:12.663 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:12.663 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:12.663 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:12.663 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:12.664 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:12.664 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:12.664 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:12.664 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:12.664 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:12.665 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:12.665 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:12.665 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:12.665 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:12.665 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:12.665 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:12.666 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:12.666 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:12.666 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:12.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:12.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:12.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:12.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:12.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:12.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:12.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:12.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:12.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:13.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:13.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:13.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:13.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:13.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:13.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:13.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:13.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:13.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:13.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:13.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:13.652 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:13.653 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:14.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27390399 2023-11-02T09:37:14.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27390399 '!=' null 2023-11-02T09:37:14.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:14.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:14.246 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:14.246 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:14.323 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:14.323 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:14.323 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:14.324 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:14.324 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:14.324 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:14.324 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:14.324 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:14.324 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:14.325 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:14.325 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:14.325 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:14.325 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:14.325 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:14.326 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:14.326 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:14.326 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:14.326 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:14.326 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:14.326 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:14.327 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:14.327 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:14.327 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:14.327 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:14.327 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:14.328 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:14.328 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:14.328 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:14.328 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:14.328 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:14.328 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:14.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:14.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:14.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:14.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:14.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:14.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:14.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:14.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:14.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:14.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:14.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:14.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:14.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:15.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:15.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:15.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:15.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:15.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:15.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:15.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:15.429 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:15.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:16.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18346786 2023-11-02T09:37:16.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18346786 '!=' null 2023-11-02T09:37:16.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:16.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:16.047 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:16.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:16.122 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:16.122 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:16.123 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:16.123 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:16.123 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:16.123 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:16.123 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:16.124 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:16.124 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:16.124 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:16.124 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:16.124 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:16.124 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:16.125 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:16.125 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:16.125 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:16.125 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:16.125 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:16.126 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:16.126 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:16.126 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:16.126 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:16.126 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:16.127 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:16.127 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:16.127 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:16.127 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:16.127 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:16.127 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:16.128 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:16.128 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:16.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:16.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:16.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:16.231 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:16.231 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:16.231 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:16.231 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:16.232 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:16.232 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:16.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:16.600 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:16.600 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:16.600 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:17.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:17.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:17.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:17.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:17.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:17.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:17.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:17.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:17.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:17.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18346668 2023-11-02T09:37:17.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18346668 '!=' null 2023-11-02T09:37:17.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:17.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:17.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:17.796 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:17.871 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:17.871 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:17.872 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:17.872 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:17.872 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:17.872 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:17.872 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:17.873 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:17.873 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:17.873 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:17.873 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:17.873 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:17.874 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:17.874 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:17.877 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:17.877 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:17.877 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:17.878 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:17.878 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:17.878 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:17.879 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:17.879 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:17.879 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:17.879 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:17.879 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:17.880 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:17.880 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:17.880 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:17.880 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:17.880 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:17.881 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:17.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:17.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:17.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:17.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:17.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:17.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:17.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:17.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:17.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:18.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:18.371 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:18.372 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:18.372 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:19.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:19.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:19.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:19.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:19.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:19.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:19.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:19.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:19.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:19.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209413 2023-11-02T09:37:19.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209413 '!=' null 2023-11-02T09:37:19.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:19.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:19.566 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:19.566 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:19.642 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:19.642 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:19.642 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:19.642 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:19.643 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:19.643 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:19.643 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:19.643 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:19.643 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:19.643 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:19.644 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:19.644 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:19.644 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:19.644 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:19.644 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:19.645 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:19.645 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:19.645 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:19.645 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:19.645 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:19.646 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:19.646 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:19.646 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:19.646 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:19.646 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:19.646 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:19.647 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:19.647 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:19.647 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:19.647 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:19.647 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:19.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:19.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:19.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:19.751 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:19.751 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:19.751 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:19.752 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:19.752 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:19.752 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:20.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:20.163 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:20.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:20.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:20.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:20.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:20.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:20.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:20.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:20.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:20.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:20.786 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:20.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:21.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472999 2023-11-02T09:37:21.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472999 '!=' null 2023-11-02T09:37:21.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:21.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:21.434 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:21.434 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:21.511 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:21.511 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:21.511 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:21.511 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:21.512 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:21.512 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:21.512 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:21.512 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:21.513 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:21.513 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:21.513 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:21.513 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:21.513 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:21.514 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:21.514 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:21.514 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:21.514 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:21.514 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:21.515 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:21.515 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:21.515 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:21.515 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:21.515 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:21.516 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:21.516 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:21.516 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:21.516 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:21.516 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:21.517 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:21.517 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:21.517 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:21.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:21.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:21.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:21.620 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:21.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:21.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:21.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:21.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:21.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:21.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:21.985 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:21.986 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:21.986 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:22.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:22.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:22.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:22.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:22.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:22.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:22.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:22.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:22.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:23.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209307 2023-11-02T09:37:23.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209307 '!=' null 2023-11-02T09:37:23.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:23.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:23.240 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:23.240 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:23.316 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:23.317 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:23.317 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:23.317 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:23.317 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:23.317 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:23.318 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:23.318 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:23.318 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:23.318 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:23.318 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:23.319 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:23.319 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:23.319 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:23.319 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:23.319 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:23.319 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:23.320 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:23.320 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:23.320 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:23.320 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:23.320 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:23.321 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:23.321 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:23.321 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:23.321 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:23.321 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:23.322 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:23.322 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:23.322 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:23.322 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:23.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:23.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:23.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:23.425 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:23.426 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:23.426 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:23.426 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:23.426 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:23.426 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:23.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:23.790 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:23.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:23.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:24.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:24.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:24.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:24.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:24.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:24.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:24.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:24.336 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:24.337 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:24.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599186 2023-11-02T09:37:24.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599186 '!=' null 2023-11-02T09:37:24.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:24.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:24.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:24.964 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:25.041 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:25.042 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:25.042 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:25.042 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:25.042 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:25.042 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:25.043 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:25.043 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:25.043 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:25.043 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:25.043 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:25.043 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:25.044 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:25.044 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:25.044 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:25.044 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:25.044 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:25.045 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:25.045 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:25.045 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:25.045 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:25.045 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:25.045 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:25.046 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:25.046 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:25.046 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:25.046 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:25.046 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:25.047 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:25.047 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:25.047 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:25.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:25.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:25.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:25.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:25.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:25.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:25.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:25.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:25.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:25.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:25.528 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:25.528 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:25.528 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:26.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:26.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:26.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:26.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:26.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:26.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:26.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:26.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:26.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:26.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599234 2023-11-02T09:37:26.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599234 '!=' null 2023-11-02T09:37:26.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:26.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:26.836 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:26.836 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:26.912 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:26.913 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:26.913 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:26.913 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:26.913 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:26.913 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:26.914 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:26.914 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:26.914 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:26.914 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:26.914 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:26.915 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:26.915 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:26.915 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:26.915 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:26.915 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:26.915 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:26.916 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:26.916 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:26.916 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:26.916 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:26.916 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:26.917 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:26.917 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:26.917 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:26.917 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:26.917 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:26.917 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:26.918 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:26.918 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:26.918 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:26.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:27.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:27.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:27.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:27.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:27.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:27.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:27.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:27.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:27.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:27.471 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:27.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:27.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:27.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:27.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:27.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:27.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:27.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:27.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:27.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:27.987 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:27.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:28.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599234 2023-11-02T09:37:28.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599234 '!=' null 2023-11-02T09:37:28.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:28.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:28.541 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:28.541 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:28.616 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:28.617 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:28.617 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:28.617 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:28.617 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:28.617 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:28.618 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:28.618 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:28.618 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:28.618 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:28.618 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:28.619 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:28.619 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:28.619 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:28.619 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:28.620 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:28.620 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:28.620 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:28.620 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:28.620 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:28.621 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:28.621 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:28.621 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:28.621 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:28.621 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:28.622 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:28.622 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:28.622 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:28.623 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:28.623 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:28.623 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:28.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:28.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:28.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:28.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:28.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:28.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:28.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:28.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:28.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:29.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:29.098 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:29.098 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:29.098 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:29.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:29.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:29.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:29.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:29.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:29.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:29.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:29.605 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:29.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:30.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25335905 2023-11-02T09:37:30.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25335905 '!=' null 2023-11-02T09:37:30.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:30.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:30.246 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:30.246 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:30.323 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:30.323 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:30.323 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:30.323 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:30.324 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:30.324 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:30.324 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:30.324 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:30.324 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:30.325 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:30.325 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:30.325 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:30.325 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:30.325 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:30.326 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:30.326 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:30.326 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:30.326 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:30.326 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:30.327 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:30.327 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:30.327 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:30.327 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:30.328 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:30.328 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:30.328 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:30.328 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:30.328 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:30.329 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:30.329 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:30.329 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:30.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:30.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:30.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:30.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:30.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:30.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:30.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:30.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:30.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:30.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:30.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:30.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:30.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:31.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:31.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:31.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:31.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:31.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:31.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:31.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:31.445 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:31.445 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:32.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473175 2023-11-02T09:37:32.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473175 '!=' null 2023-11-02T09:37:32.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:32.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:32.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:32.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:32.132 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:32.132 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:32.133 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:32.133 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:32.133 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:32.133 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:32.133 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:32.134 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:32.134 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:32.134 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:32.134 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:32.134 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:32.135 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:32.135 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:32.135 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:32.135 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:32.136 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:32.136 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:32.136 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:32.136 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:32.136 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:32.137 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:32.137 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:32.137 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:32.137 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:32.137 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:32.137 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:32.138 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:32.138 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:32.138 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:32.138 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:32.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:32.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:32.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:32.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:32.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:32.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:32.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:32.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:32.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:32.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:32.609 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:32.609 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:32.610 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:33.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:33.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:33.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:33.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:33.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:33.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:33.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:33.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:33.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:33.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209867 2023-11-02T09:37:33.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209867 '!=' null 2023-11-02T09:37:33.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:33.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:33.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:33.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:33.871 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:33.871 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:33.871 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:33.871 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:33.872 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:33.872 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:33.872 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:33.872 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:33.872 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:33.873 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:33.873 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:33.873 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:33.873 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:33.873 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:33.874 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:33.874 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:33.874 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:33.874 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:33.874 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:33.877 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:33.877 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:33.877 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:33.877 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:33.878 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:33.880 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:33.880 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:33.880 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:33.880 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:33.881 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:33.881 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:33.881 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:33.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:33.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:33.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:33.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:33.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:33.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:33.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:33.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:33.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:34.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:34.347 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:34.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:34.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:34.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:34.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:34.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:34.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:34.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:34.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:34.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:34.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:34.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:35.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473375 2023-11-02T09:37:35.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473375 '!=' null 2023-11-02T09:37:35.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:35.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:35.475 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:35.475 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:35.552 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:35.552 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:35.552 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:35.553 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:35.553 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:35.553 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:35.553 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:35.553 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:35.554 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:35.554 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:35.554 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:35.554 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:35.554 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:35.554 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:35.555 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:35.555 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:35.555 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:35.555 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:35.556 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:35.556 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:35.556 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:35.556 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:35.556 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:35.557 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:35.557 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:35.557 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:35.557 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:35.559 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:35.559 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:35.559 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:35.559 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:35.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:35.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:35.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:35.661 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:35.661 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:35.661 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:35.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:35.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:35.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:36.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:36.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:36.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:36.044 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:36.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:36.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:36.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:36.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:36.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:36.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:36.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:36.551 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:36.551 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:37.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473431 2023-11-02T09:37:37.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473431 '!=' null 2023-11-02T09:37:37.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:37.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:37.334 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:37.335 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:37.410 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:37.411 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:37.411 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:37.411 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:37.411 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:37.412 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:37.412 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:37.412 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:37.412 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:37.412 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:37.413 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:37.413 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:37.413 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:37.413 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:37.413 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:37.414 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:37.414 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:37.414 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:37.414 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:37.414 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:37.414 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:37.415 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:37.415 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:37.415 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:37.415 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:37.415 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:37.416 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:37.416 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:37.416 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:37.416 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:37.416 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:37.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:37.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:37.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:37.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:37.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:37.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:37.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:37.521 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:37.521 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:37.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:37.885 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:37.885 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:37.886 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:38.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:38.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:38.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:38.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:38.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:38.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:38.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:38.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:38.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:38.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210281 2023-11-02T09:37:38.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210281 '!=' null 2023-11-02T09:37:38.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:38.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:38.991 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:38.991 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:39.067 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:39.068 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:39.068 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:39.068 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:39.068 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:39.068 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:39.069 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:39.069 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:39.069 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:39.069 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:39.070 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:39.070 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:39.070 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:39.070 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:39.070 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:39.071 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:39.071 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:39.071 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:39.072 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:39.072 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:39.072 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:39.073 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:39.073 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:39.073 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:39.074 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:39.074 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:39.074 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:39.074 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:39.074 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:39.074 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:39.075 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:39.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:39.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:39.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:39.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:39.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:39.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:39.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:39.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:39.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:39.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:39.620 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:39.620 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:39.620 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:40.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:40.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:40.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:40.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:40.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:40.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:40.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:40.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:40.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:40.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473507 2023-11-02T09:37:40.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473507 '!=' null 2023-11-02T09:37:40.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:40.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:40.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:40.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:40.880 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:40.881 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:40.881 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:40.881 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:40.881 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:40.881 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:40.883 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:40.883 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:40.883 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:40.883 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:40.884 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:40.884 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:40.884 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:40.884 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:40.884 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:40.885 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:40.885 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:40.885 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:40.885 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:40.885 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:40.885 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:40.886 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:40.886 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:40.886 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:40.886 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:40.886 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:40.887 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:40.887 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:40.887 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:40.887 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:40.887 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:40.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:40.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:40.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:40.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:40.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:40.990 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:40.990 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:40.990 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:40.990 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:41.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:41.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:41.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:41.353 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:41.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:41.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:41.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:41.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:41.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:41.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:41.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:41.903 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:41.903 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:42.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473507 2023-11-02T09:37:42.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473507 '!=' null 2023-11-02T09:37:42.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:42.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:42.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:42.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:42.727 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:42.727 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:42.727 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:42.728 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:42.728 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:42.728 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:42.728 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:42.728 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:42.728 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:42.729 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:42.729 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:42.729 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:42.729 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:42.729 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:42.730 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:42.730 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:42.730 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:42.730 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:42.730 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:42.731 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:42.731 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:42.731 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:42.731 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:42.731 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:42.731 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:42.732 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:42.732 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:42.732 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:42.732 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:42.732 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:42.733 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:42.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:42.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:42.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:42.836 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:42.836 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:42.836 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:42.836 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:42.837 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:42.837 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:43.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:43.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:43.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:43.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:43.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:43.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:43.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:43.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:43.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:43.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:43.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:43.739 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:43.739 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:44.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210187 2023-11-02T09:37:44.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210187 '!=' null 2023-11-02T09:37:44.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:44.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:44.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:44.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:44.322 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:44.322 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:44.322 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:44.323 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:44.323 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:44.323 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:44.323 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:44.323 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:44.324 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:44.324 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:44.324 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:44.324 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:44.324 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:44.324 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:44.325 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:44.325 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:44.325 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:44.325 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:44.325 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:44.326 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:44.326 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:44.326 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:44.327 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:44.327 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:44.327 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:44.327 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:44.327 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:44.328 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:44.328 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:44.328 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:44.328 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:44.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:44.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:44.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:44.431 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:44.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:44.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:44.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:44.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:44.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:44.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:44.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:44.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:44.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:45.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:45.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:45.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:45.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:45.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:45.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:45.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:45.334 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:45.334 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:45.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473411 2023-11-02T09:37:45.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473411 '!=' null 2023-11-02T09:37:45.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:45.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:45.898 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:45.898 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:45.975 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:45.976 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:45.976 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:45.976 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:45.976 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:45.976 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:45.977 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:45.977 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:45.977 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:45.977 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:45.977 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:45.978 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:45.978 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:45.978 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:45.978 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:45.978 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:45.979 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:45.979 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:45.979 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:45.979 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:45.980 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:45.980 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:45.980 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:45.980 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:45.980 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:45.980 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:45.981 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:45.981 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:45.981 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:45.981 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:45.981 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:45.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:46.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:46.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:46.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:46.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:46.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:46.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:46.086 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:46.086 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:46.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:46.490 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:46.490 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:46.491 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:47.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:47.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:47.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:47.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:47.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:47.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:47.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:47.071 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:47.071 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:47.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473423 2023-11-02T09:37:47.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473423 '!=' null 2023-11-02T09:37:47.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:47.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:47.703 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:47.703 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:47.781 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:47.782 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:47.782 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:47.782 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:47.783 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:47.783 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:47.783 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:47.783 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:47.783 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:47.784 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:47.784 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:47.784 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:47.784 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:47.784 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:47.785 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:47.785 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:47.785 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:47.785 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:47.785 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:47.786 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:47.786 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:47.786 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:47.786 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:47.786 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:47.787 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:47.787 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:47.787 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:47.787 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:47.787 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:47.788 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:47.788 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:47.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:47.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:47.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:47.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:47.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:47.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:47.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:47.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:47.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:48.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:48.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:48.256 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:48.256 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:48.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:48.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:48.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:48.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:48.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:48.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:48.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:48.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:48.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:49.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210145 2023-11-02T09:37:49.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210145 '!=' null 2023-11-02T09:37:49.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:49.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:49.409 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:49.409 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:49.485 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:49.486 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:49.486 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:49.486 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:49.486 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:49.487 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:49.487 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:49.487 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:49.487 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:49.487 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:49.488 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:49.488 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:49.488 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:49.489 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:49.489 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:49.489 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:49.489 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:49.489 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:49.490 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:49.491 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:49.491 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:49.492 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:49.492 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:49.492 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:49.492 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:49.493 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:49.493 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:49.493 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:49.493 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:49.493 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:49.494 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:49.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:49.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:49.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:49.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:49.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:49.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:49.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:49.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:49.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:49.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:49.993 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:49.994 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:49.994 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:50.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:50.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:50.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:50.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:50.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:50.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:50.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:50.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:50.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:51.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473428 2023-11-02T09:37:51.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473428 '!=' null 2023-11-02T09:37:51.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:51.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:51.222 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:51.223 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:51.300 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:51.300 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:51.300 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:51.301 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:51.301 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:51.301 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:51.301 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:51.301 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:51.302 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:51.302 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:51.302 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:51.302 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:51.302 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:51.303 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:51.303 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:51.303 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:51.303 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:51.303 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:51.304 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:51.304 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:51.304 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:51.304 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:51.304 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:51.304 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:51.305 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:51.305 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:51.305 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:51.305 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:51.305 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:51.306 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:51.306 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:51.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:51.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:51.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:51.409 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:51.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:51.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:51.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:51.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:51.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:51.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:51.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:51.781 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:51.781 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:52.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:52.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:52.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:52.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:52.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:52.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:52.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:52.335 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:52.336 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:52.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210184 2023-11-02T09:37:52.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210184 '!=' null 2023-11-02T09:37:52.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:52.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:52.912 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:52.912 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:52.989 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:52.989 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:52.989 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:52.990 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:52.990 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:52.990 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:52.990 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:52.991 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:52.991 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:52.992 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:52.992 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:52.992 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:52.992 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:52.992 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:52.993 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:52.993 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:52.993 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:52.993 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:52.993 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:52.994 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:52.994 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:52.994 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:52.994 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:52.994 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:52.994 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:52.995 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:52.995 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:52.995 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:52.995 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:52.995 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:52.996 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:52.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:53.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:53.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:53.101 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:53.101 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:53.101 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:53.102 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:53.102 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:53.102 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:53.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:53.503 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:53.503 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:53.504 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:54.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:54.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:54.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:54.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:54.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:54.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:54.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:54.119 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:54.119 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:54.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473432 2023-11-02T09:37:54.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473432 '!=' null 2023-11-02T09:37:54.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:54.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:54.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:54.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:54.845 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:54.845 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:54.846 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:54.846 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:54.846 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:54.846 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:54.846 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:54.847 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:54.847 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:54.847 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:54.847 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:54.847 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:54.848 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:54.848 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:54.848 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:54.848 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:54.848 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:54.849 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:54.849 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:54.849 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:54.849 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:54.849 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:54.849 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:54.850 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:54.850 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:54.850 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:54.850 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:54.850 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:54.851 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:54.851 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:54.851 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:54.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:54.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:54.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:54.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:54.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:54.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:54.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:54.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:54.956 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:55.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:55.427 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:55.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:55.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:55.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:55.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:55.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:55.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:55.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:55.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:55.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:55.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:55.950 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:56.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473432 2023-11-02T09:37:56.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473432 '!=' null 2023-11-02T09:37:56.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:56.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:56.618 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:56.619 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:56.692 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:56.692 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:56.692 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:56.693 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:56.693 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:56.693 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:56.693 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:56.693 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:56.694 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:56.694 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:56.694 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:56.694 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:56.694 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:56.695 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:56.695 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:56.695 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:56.695 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:56.695 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:56.696 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:56.696 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:56.696 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:56.696 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:56.696 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:56.697 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:56.697 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:56.697 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:56.697 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:56.697 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:56.698 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:56.698 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:56.698 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:56.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:56.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:56.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:56.801 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:56.802 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:56.802 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:56.802 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:56.802 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:56.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:57.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:57.174 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:57.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:57.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:57.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:57.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:57.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:57.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:57.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:57.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:57.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:57.758 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:57.758 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:37:58.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473381 2023-11-02T09:37:58.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473381 '!=' null 2023-11-02T09:37:58.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:37:58.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:58.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:37:58.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:37:58.391 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:37:58.391 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:37:58.392 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:58.392 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:58.392 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:37:58.393 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:58.393 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:37:58.393 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:37:58.393 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:58.394 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:37:58.394 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:58.394 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:58.394 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:58.395 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:58.395 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:58.395 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:37:58.395 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:37:58.395 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:37:58.396 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:37:58.399 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:37:58.399 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:37:58.400 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:58.400 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:37:58.400 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:37:58.400 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:37:58.400 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:37:58.401 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:37:58.401 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:37:58.402 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:37:58.402 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:37:58.402 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:37:58.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:37:58.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:37:58.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:37:58.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:37:58.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:37:58.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:37:58.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:37:58.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:37:58.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:37:58.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:37:58.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:37:58.930 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:37:58.930 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:37:59.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:37:59.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:37:59.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:37:59.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:37:59.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:37:59.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:37:59.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:37:59.466 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:37:59.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:00.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210117 2023-11-02T09:38:00.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210117 '!=' null 2023-11-02T09:38:00.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:00.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:00.110 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:00.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:00.188 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:00.189 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:00.189 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:00.189 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:00.189 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:00.190 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:00.190 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:00.190 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:00.190 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:00.190 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:00.191 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:00.191 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:00.191 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:00.191 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:00.191 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:00.192 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:00.192 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:00.192 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:00.192 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:00.192 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:00.193 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:00.193 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:00.193 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:00.193 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:00.193 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:00.194 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:00.194 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:00.194 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:00.194 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:00.194 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:00.195 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:00.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:00.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:00.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:00.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:00.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:00.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:00.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:00.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:00.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:00.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:00.696 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:00.696 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:00.697 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:01.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:01.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:01.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:01.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:01.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:01.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:01.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:01.367 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:01.368 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:01.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473244 2023-11-02T09:38:01.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473244 '!=' null 2023-11-02T09:38:01.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:01.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:01.964 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:01.964 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:02.042 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:02.043 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:02.043 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:02.043 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:02.043 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:02.044 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:02.044 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:02.044 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:02.044 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:02.045 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:02.045 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:02.045 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:02.045 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:02.045 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:02.046 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:02.046 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:02.046 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:02.047 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:02.047 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:02.047 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:02.048 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:02.048 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:02.048 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:02.048 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:02.049 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:02.049 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:02.049 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:02.049 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:02.050 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:02.050 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:02.050 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:02.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:02.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:02.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:02.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:02.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:02.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:02.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:02.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:02.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:02.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:02.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:02.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:02.556 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:03.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:03.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:03.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:03.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:03.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:03.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:03.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:03.167 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:03.167 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:03.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209636 2023-11-02T09:38:03.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209636 '!=' null 2023-11-02T09:38:03.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:03.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:03.826 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:03.826 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:03.902 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:03.902 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:03.903 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:03.903 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:03.903 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:03.903 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:03.903 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:03.904 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:03.904 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:03.904 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:03.905 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:03.905 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:03.905 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:03.905 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:03.905 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:03.906 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:03.906 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:03.906 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:03.906 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:03.906 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:03.907 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:03.907 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:03.907 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:03.907 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:03.907 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:03.908 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:03.908 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:03.908 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:03.908 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:03.908 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:03.910 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:03.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:04.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:04.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:04.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:04.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:04.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:04.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:04.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:04.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:04.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:04.375 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:04.376 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:04.376 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:05.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:05.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:05.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:05.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:05.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:05.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:05.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:05.172 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:05.172 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:05.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473066 2023-11-02T09:38:05.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473066 '!=' null 2023-11-02T09:38:05.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:05.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:05.703 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:05.703 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:05.779 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:05.780 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:05.780 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:05.780 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:05.780 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:05.781 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:05.781 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:05.781 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:05.781 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:05.782 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:05.782 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:05.782 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:05.782 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:05.782 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:05.783 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:05.783 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:05.783 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:05.783 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:05.783 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:05.783 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:05.784 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:05.784 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:05.784 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:05.784 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:05.785 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:05.785 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:05.785 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:05.785 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:05.785 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:05.786 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:05.786 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:05.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:05.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:05.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:05.889 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:05.889 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:05.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:05.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:05.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:05.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:06.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:06.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:06.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:06.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:06.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:06.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:06.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:06.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:06.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:06.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:06.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:06.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:06.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:07.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472909 2023-11-02T09:38:07.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472909 '!=' null 2023-11-02T09:38:07.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:07.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:07.355 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:07.355 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:07.435 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:07.436 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:07.436 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:07.436 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:07.436 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:07.436 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:07.437 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:07.437 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:07.437 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:07.437 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:07.438 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:07.438 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:07.438 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:07.438 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:07.439 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:07.439 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:07.439 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:07.439 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:07.439 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:07.440 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:07.440 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:07.440 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:07.440 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:07.440 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:07.441 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:07.441 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:07.441 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:07.441 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:07.441 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:07.442 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:07.442 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:07.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:07.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:07.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:07.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:07.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:07.543 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:07.543 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:07.543 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:07.543 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:07.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:07.911 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:07.912 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:07.912 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:08.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:08.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:08.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:08.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:08.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:08.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:08.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:08.519 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:08.519 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:09.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209445 2023-11-02T09:38:09.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209445 '!=' null 2023-11-02T09:38:09.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:09.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:09.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:09.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:09.099 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:09.099 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:09.100 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:09.100 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:09.100 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:09.100 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:09.101 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:09.101 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:09.101 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:09.101 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:09.101 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:09.101 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:09.102 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:09.102 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:09.103 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:09.104 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:09.104 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:09.104 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:09.104 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:09.104 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:09.105 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:09.108 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:09.109 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:09.109 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:09.110 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:09.110 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:09.110 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:09.111 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:09.111 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:09.111 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:09.111 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:09.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:09.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:09.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:09.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:09.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:09.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:09.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:09.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:09.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:09.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:09.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:09.597 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:09.597 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:10.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:10.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:10.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:10.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:10.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:10.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:10.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:10.101 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:10.101 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:10.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472972 2023-11-02T09:38:10.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472972 '!=' null 2023-11-02T09:38:10.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:10.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:10.614 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:10.614 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:10.692 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:10.693 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:10.693 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:10.695 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:10.695 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:10.695 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:10.695 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:10.695 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:10.696 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:10.696 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:10.696 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:10.696 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:10.696 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:10.697 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:10.697 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:10.697 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:10.697 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:10.697 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:10.698 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:10.698 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:10.698 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:10.698 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:10.698 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:10.699 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:10.699 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:10.699 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:10.699 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:10.704 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:10.704 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:10.704 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:10.704 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:10.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:10.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:10.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:10.802 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:10.802 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:10.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:10.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:10.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:10.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:11.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:11.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:11.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:11.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:11.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:11.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:11.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:11.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:11.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:11.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:11.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:11.879 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:11.879 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:12.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472972 2023-11-02T09:38:12.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472972 '!=' null 2023-11-02T09:38:12.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:12.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:12.405 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:12.405 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:12.481 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:12.482 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:12.482 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:12.482 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:12.482 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:12.483 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:12.483 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:12.483 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:12.483 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:12.483 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:12.484 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:12.484 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:12.484 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:12.484 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:12.484 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:12.484 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:12.485 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:12.485 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:12.485 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:12.485 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:12.485 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:12.486 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:12.486 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:12.486 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:12.486 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:12.486 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:12.487 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:12.487 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:12.487 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:12.487 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:12.487 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:12.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:12.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:12.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:12.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:12.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:12.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:12.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:12.592 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:12.592 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:12.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:12.958 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:12.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:12.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:13.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:13.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:13.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:13.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:13.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:13.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:13.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:13.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:13.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:14.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209714 2023-11-02T09:38:14.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209714 '!=' null 2023-11-02T09:38:14.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:14.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:14.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:14.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:14.154 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:14.154 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:14.154 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:14.155 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:14.155 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:14.155 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:14.155 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:14.155 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:14.156 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:14.156 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:14.156 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:14.156 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:14.156 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:14.157 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:14.157 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:14.157 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:14.157 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:14.157 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:14.158 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:14.158 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:14.160 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:14.160 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:14.160 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:14.160 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:14.161 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:14.161 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:14.161 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:14.161 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:14.161 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:14.161 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:14.162 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:14.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:14.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:14.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:14.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:14.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:14.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:14.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:14.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:14.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:14.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:14.635 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:14.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:14.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:15.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:15.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:15.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:15.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:15.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:15.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:15.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:15.150 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:15.150 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:15.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473219 2023-11-02T09:38:15.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473219 '!=' null 2023-11-02T09:38:15.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:15.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:15.900 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:15.900 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:15.977 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:15.978 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:15.978 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:15.978 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:15.978 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:15.979 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:15.979 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:15.979 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:15.979 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:15.980 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:15.980 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:15.980 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:15.980 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:15.980 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:15.980 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:15.981 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:15.981 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:15.981 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:15.981 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:15.981 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:15.982 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:15.982 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:15.987 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:15.987 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:15.987 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:15.987 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:15.988 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:15.988 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:15.988 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:15.989 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:15.989 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:15.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:16.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:16.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:16.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:16.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:16.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:16.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:16.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:16.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:16.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:16.527 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:16.527 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:16.528 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:17.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:17.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:17.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:17.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:17.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:17.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:17.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:17.046 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:17.047 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:17.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473222 2023-11-02T09:38:17.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473222 '!=' null 2023-11-02T09:38:17.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:17.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:17.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:17.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:17.675 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:17.675 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:17.675 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:17.675 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:17.676 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:17.676 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:17.676 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:17.676 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:17.676 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:17.677 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:17.677 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:17.677 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:17.677 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:17.677 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:17.678 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:17.678 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:17.678 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:17.678 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:17.678 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:17.678 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:17.679 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:17.679 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:17.679 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:17.679 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:17.679 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:17.680 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:17.680 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:17.680 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:17.680 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:17.680 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:17.681 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:17.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:17.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:17.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:17.783 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:17.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:17.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:17.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:17.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:17.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:18.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:18.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:18.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:18.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:18.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:18.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:18.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:18.688 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:18.688 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:19.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336420 2023-11-02T09:38:19.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336420 '!=' null 2023-11-02T09:38:19.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:19.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:19.233 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:19.233 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:19.309 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:19.310 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:19.310 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:19.310 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:19.310 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:19.310 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:19.311 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:19.311 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:19.311 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:19.311 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:19.311 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:19.312 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:19.312 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:19.312 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:19.312 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:19.312 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:19.312 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:19.313 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:19.313 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:19.313 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:19.313 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:19.313 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:19.314 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:19.314 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:19.314 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:19.314 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:19.314 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:19.315 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:19.315 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:19.315 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:19.315 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:19.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:19.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:19.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:19.418 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:19.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:19.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:19.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:19.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:19.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:19.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:19.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:19.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:19.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:20.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:20.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:20.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:20.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:20.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:20.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:20.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:20.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:20.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:20.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599707 2023-11-02T09:38:20.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599707 '!=' null 2023-11-02T09:38:20.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:20.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:20.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:20.974 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:21.050 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:21.050 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:21.050 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:21.051 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:21.051 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:21.051 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:21.051 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:21.051 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:21.052 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:21.052 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:21.052 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:21.052 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:21.053 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:21.053 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:21.053 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:21.053 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:21.053 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:21.053 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:21.054 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:21.054 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:21.054 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:21.054 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:21.054 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:21.055 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:21.055 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:21.055 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:21.055 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:21.055 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:21.056 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:21.056 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:21.056 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:21.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:21.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:21.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:21.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:21.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:21.160 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:21.160 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:21.160 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:21.160 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:21.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:21.531 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:21.532 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:21.532 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:22.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:22.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:22.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:22.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:22.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:22.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599707 2023-11-02T09:38:22.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599707 '!=' null 2023-11-02T09:38:22.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:22.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:22.617 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:22.617 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:22.694 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:22.694 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:22.694 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:22.695 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:22.695 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:22.695 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:22.695 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:22.696 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:22.696 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:22.696 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:22.696 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:22.697 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:22.697 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:22.697 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:22.697 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:22.697 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:22.698 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:22.698 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:22.698 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:22.698 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:22.698 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:22.699 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:22.699 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:22.701 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:22.701 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:22.702 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:22.702 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:22.702 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:22.702 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:22.704 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:22.705 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:22.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:22.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:22.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:22.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:22.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:22.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:22.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:22.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:22.805 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:23.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:23.244 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:23.244 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:23.244 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:23.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:23.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:23.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:23.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:23.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:23.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:23.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:23.779 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:23.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:24.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336402 2023-11-02T09:38:24.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336402 '!=' null 2023-11-02T09:38:24.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:24.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:24.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:24.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:24.457 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:24.457 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:24.457 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:24.458 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:24.458 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:24.458 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:24.458 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:24.458 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:24.458 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:24.459 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:24.459 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:24.459 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:24.459 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:24.459 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:24.460 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:24.460 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:24.460 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:24.460 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:24.460 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:24.461 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:24.461 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:24.461 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:24.461 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:24.461 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:24.461 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:24.462 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:24.462 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:24.462 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:24.462 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:24.462 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:24.463 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:24.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:24.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:24.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:24.566 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:24.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:24.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:24.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:24.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:24.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:24.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:24.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:24.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:24.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:25.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:25.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:25.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:25.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:25.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:25.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:25.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:25.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:25.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:26.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599698 2023-11-02T09:38:26.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599698 '!=' null 2023-11-02T09:38:26.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:26.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:26.190 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:26.190 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:26.266 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:26.266 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:26.267 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:26.267 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:26.267 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:26.267 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:26.267 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:26.268 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:26.268 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:26.268 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:26.268 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:26.268 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:26.268 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:26.269 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:26.269 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:26.269 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:26.269 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:26.269 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:26.270 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:26.270 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:26.270 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:26.273 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:26.273 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:26.273 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:26.273 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:26.274 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:26.274 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:26.274 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:26.275 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:26.275 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:26.275 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:26.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:26.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:26.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:26.376 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:26.376 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:26.376 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:26.376 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:26.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:26.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:26.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:26.744 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:26.744 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:26.745 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:27.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:27.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:27.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:27.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:27.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:27.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:27.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:27.381 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:27.381 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:27.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599758 2023-11-02T09:38:27.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599758 '!=' null 2023-11-02T09:38:27.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:27.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:27.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:27.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:27.960 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:27.961 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:27.961 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:27.961 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:27.961 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:27.962 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:27.962 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:27.962 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:27.962 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:27.962 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:27.963 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:27.963 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:27.963 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:27.963 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:27.963 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:27.963 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:27.964 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:27.964 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:27.964 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:27.965 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:27.965 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:27.965 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:27.965 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:27.966 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:27.966 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:27.966 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:27.966 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:27.966 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:27.967 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:27.967 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:27.967 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:27.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:28.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:28.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:28.071 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:28.071 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:28.072 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:28.072 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:28.072 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:28.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:28.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:28.523 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:28.523 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:28.524 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:29.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:29.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:29.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:29.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:29.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:29.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:29.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:29.154 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:29.154 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:29.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336437 2023-11-02T09:38:29.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336437 '!=' null 2023-11-02T09:38:29.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:29.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:29.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:29.830 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:29.905 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:29.906 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:29.906 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:29.906 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:29.906 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:29.906 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:29.907 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:29.907 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:29.907 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:29.907 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:29.907 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:29.907 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:29.908 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:29.908 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:29.908 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:29.908 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:29.908 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:29.909 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:29.909 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:29.909 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:29.909 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:29.909 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:29.909 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:29.910 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:29.910 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:29.910 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:29.910 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:29.910 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:29.911 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:29.911 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:29.911 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:29.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:30.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:30.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:30.015 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:30.015 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:30.015 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:30.015 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:30.015 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:30.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:30.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:30.379 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:30.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:30.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:31.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:31.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:31.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:31.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:31.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:31.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:31.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:31.018 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:31.018 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:31.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473391 2023-11-02T09:38:31.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473391 '!=' null 2023-11-02T09:38:31.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:31.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:31.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:31.703 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:31.779 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:31.780 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:31.780 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:31.780 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:31.780 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:31.780 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:31.781 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:31.781 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:31.781 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:31.781 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:31.782 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:31.782 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:31.782 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:31.782 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:31.783 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:31.783 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:31.783 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:31.783 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:31.784 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:31.784 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:31.784 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:31.784 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:31.784 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:31.785 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:31.785 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:31.786 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:31.786 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:31.787 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:31.787 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:31.787 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:31.788 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:31.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:31.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:31.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:31.889 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:31.889 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:31.889 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:31.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:31.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:31.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:32.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:32.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:32.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:32.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:32.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:32.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:32.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:32.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:32.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:32.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:32.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:32.797 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:32.797 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:33.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210117 2023-11-02T09:38:33.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210117 '!=' null 2023-11-02T09:38:33.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:33.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:33.364 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:33.364 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:33.439 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:33.439 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:33.440 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:33.440 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:33.440 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:33.440 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:33.440 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:33.441 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:33.441 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:33.441 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:33.441 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:33.441 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:33.442 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:33.442 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:33.442 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:33.442 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:33.442 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:33.442 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:33.443 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:33.443 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:33.443 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:33.443 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:33.443 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:33.444 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:33.444 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:33.444 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:33.444 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:33.444 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:33.445 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:33.445 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:33.445 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:33.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:33.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:33.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:33.548 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:33.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:33.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:33.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:33.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:33.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:33.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:33.910 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:33.911 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:33.911 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:34.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:34.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:34.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:34.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:34.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:34.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:34.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:34.413 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:34.413 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:34.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473368 2023-11-02T09:38:34.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473368 '!=' null 2023-11-02T09:38:34.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:34.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:34.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:34.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:35.014 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:35.014 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:35.014 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:35.015 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:35.015 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:35.015 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:35.015 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:35.015 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:35.016 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:35.016 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:35.016 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:35.016 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:35.016 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:35.017 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:35.017 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:35.017 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:35.017 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:35.017 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:35.018 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:35.018 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:35.018 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:35.018 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:35.018 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:35.019 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:35.020 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:35.020 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:35.020 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:35.021 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:35.021 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:35.021 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:35.021 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:35.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:35.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:35.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:35.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:35.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:35.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:35.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:35.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:35.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:35.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:35.489 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:35.489 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:35.490 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:36.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:36.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:36.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:36.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:36.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:36.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:36.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:36.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:36.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:36.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473314 2023-11-02T09:38:36.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473314 '!=' null 2023-11-02T09:38:36.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:36.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:36.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:36.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:36.963 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:36.964 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:36.964 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:36.964 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:36.964 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:36.964 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:36.965 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:36.965 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:36.965 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:36.965 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:36.965 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:36.965 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:36.966 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:36.966 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:36.966 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:36.966 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:36.966 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:36.967 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:36.967 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:36.967 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:36.967 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:36.967 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:36.968 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:36.968 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:36.968 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:36.968 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:36.968 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:36.968 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:36.969 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:36.969 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:36.969 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:36.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:37.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:37.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:37.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:37.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:37.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:37.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:37.074 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:37.074 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:37.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:37.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:37.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:37.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:38.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:38.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:38.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:38.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:38.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:38.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:38.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:38.050 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:38.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:38.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210052 2023-11-02T09:38:38.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210052 '!=' null 2023-11-02T09:38:38.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:38.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:38.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:38.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:38.870 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:38.870 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:38.871 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:38.871 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:38.871 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:38.871 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:38.872 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:38.872 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:38.872 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:38.872 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:38.872 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:38.872 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:38.873 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:38.873 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:38.873 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:38.873 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:38.873 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:38.874 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:38.874 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:38.874 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:38.874 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:38.874 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:38.875 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:38.875 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:38.875 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:38.875 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:38.875 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:38.875 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:38.876 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:38.876 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:38.876 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:38.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:38.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:38.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:38.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:38.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:38.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:38.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:38.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:38.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:39.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:39.343 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:39.344 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:39.344 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:39.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:39.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:39.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:39.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:39.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:39.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:39.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:39.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:39.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:40.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210052 2023-11-02T09:38:40.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210052 '!=' null 2023-11-02T09:38:40.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:40.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:40.357 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:40.357 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:40.433 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:40.434 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:40.434 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:40.434 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:40.435 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [ 2023-11-02T09:38:40.435 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:40.435 INFO:tasks.workunit.client.0.smithi116.stdout: "priority": 190, 2023-11-02T09:38:40.435 INFO:tasks.workunit.client.0.smithi116.stdout: "items": [ 2023-11-02T09:38:40.435 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:40.436 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:40.436 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:40.436 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:40.436 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:40.436 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:40.437 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:40.437 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2023-11-02T09:38:40.437 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:40.437 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:40.438 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "1.0", 2023-11-02T09:38:40.438 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:40.438 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:40.438 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:40.438 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:40.439 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:40.440 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:40.440 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:40.440 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:40.441 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:40.441 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:40.441 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:40.442 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:40.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:40.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:40.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:40.543 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:40.543 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:40.544 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:40.544 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:40.544 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:40.544 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:40.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:40.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:40.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:40.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:41.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:41.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:41.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:41.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:41.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:41.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:41.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:41.622 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:41.623 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:42.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473381 2023-11-02T09:38:42.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473381 '!=' null 2023-11-02T09:38:42.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:42.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:42.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:42.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:42.465 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:42.466 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:42.466 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:42.466 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:42.466 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:42.468 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:42.468 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:42.468 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:42.468 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:42.468 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:42.469 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:42.469 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:42.469 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:42.469 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:42.469 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:42.470 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:42.471 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:42.472 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:42.472 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:42.473 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:42.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:42.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:42.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:42.575 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:42.576 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:42.576 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:42.576 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:42.577 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:42.577 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:43.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:43.248 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:43.248 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:43.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:43.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:43.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:43.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:43.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:43.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:43.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:43.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:43.906 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:43.906 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:44.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24899688 2023-11-02T09:38:44.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24899688 '!=' null 2023-11-02T09:38:44.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:44.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:44.558 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:44.558 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:44.634 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:44.634 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:44.635 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:44.635 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:44.635 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:44.635 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:44.635 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:44.636 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:44.636 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:44.636 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:44.636 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:44.636 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:44.637 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:44.637 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:44.637 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:44.637 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:44.637 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:44.638 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:44.638 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:44.638 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:44.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:44.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:44.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:44.744 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:44.744 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:44.745 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:44.745 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:44.745 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:44.745 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:45.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:45.160 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:45.161 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:45.161 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:45.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:45.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:45.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:45.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:45.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:45.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:45.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:45.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:45.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:46.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24899757 2023-11-02T09:38:46.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24899757 '!=' null 2023-11-02T09:38:46.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:46.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:46.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:46.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:46.382 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:46.382 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:46.383 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:46.383 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:46.383 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:46.383 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:46.383 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:46.384 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:46.384 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:46.384 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:46.384 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:46.384 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:46.385 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:46.385 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:46.385 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:46.385 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:46.385 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:46.385 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:46.386 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:46.386 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:46.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:46.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:46.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:46.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:46.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:46.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:46.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:46.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:46.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:46.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:38:46.967 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:46.968 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:46.968 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:47.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2023-11-02T09:38:47.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2023-11-02T09:38:47.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:47.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:47.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:47.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:47.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:47.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:47.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:48.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11794541 2023-11-02T09:38:48.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11794541 '!=' null 2023-11-02T09:38:48.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:48.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:48.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:48.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:48.151 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:48.151 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:48.151 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:48.151 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:48.152 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:48.152 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:48.152 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:48.152 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:48.152 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:48.152 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:48.153 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:48.153 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:48.153 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:48.153 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:48.153 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:48.154 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:48.154 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:48.155 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:48.155 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:48.156 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:48.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:48.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:48.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:48.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:48.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:48.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:48.261 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:48.261 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:48.261 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:48.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:38:48.652 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:48.652 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:48.652 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:49.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:38:49.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 1 2023-11-02T09:38:49.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2023-11-02T09:38:49.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2023-11-02T09:38:49.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:49.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:49.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:49.241 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:49.242 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:49.242 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:49.242 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:49.242 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:49.243 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:49.243 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:49.243 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:49.243 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:49.243 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:49.244 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:49.244 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:49.244 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:49.244 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:49.244 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:49.245 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:49.245 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:49.245 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:49.245 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:49.245 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:49.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:49.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:49.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:49.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:49.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:49.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:49.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:49.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:49.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:49.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:38:49.792 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:49.792 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:49.793 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:50.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:38:50.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:38:50.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:50.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:50.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:50.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:50.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:50.297 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:50.297 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:51.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22871452 2023-11-02T09:38:51.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22871452 '!=' null 2023-11-02T09:38:51.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:51.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:51.015 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:51.015 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:51.091 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:51.092 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:51.092 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:51.092 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:51.092 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:51.092 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:51.093 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:51.093 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:51.093 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:51.093 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:51.093 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:51.093 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:51.094 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:51.094 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:51.094 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:51.094 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:51.094 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:51.095 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:51.095 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:51.095 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:51.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:51.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:51.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:51.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:51.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:51.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:51.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:51.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:51.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:51.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:38:51.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:51.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:51.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:52.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:38:52.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:38:52.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:52.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:52.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:52.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:52.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:52.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:52.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:52.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22693740 2023-11-02T09:38:52.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22693740 '!=' null 2023-11-02T09:38:52.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:52.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:52.605 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:52.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:52.683 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:52.684 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:52.684 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:52.684 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:52.684 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:52.685 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:52.685 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:52.685 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:52.685 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:52.685 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:52.685 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:52.686 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:52.686 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:52.686 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:52.686 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:52.686 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:52.687 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:52.687 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:52.687 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:52.687 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:52.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:52.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:52.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:52.793 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:52.793 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:52.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:52.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:52.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:52.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:53.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:38:53.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:53.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:53.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:53.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:38:53.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:38:53.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:53.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:53.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:53.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:53.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:53.850 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:53.850 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:54.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22693740 2023-11-02T09:38:54.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22693740 '!=' null 2023-11-02T09:38:54.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:54.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:54.358 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:54.358 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:54.435 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:54.435 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:54.435 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:54.436 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:54.436 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:54.436 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:54.436 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:54.436 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:54.437 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:54.437 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:54.437 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:54.437 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:54.437 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:54.438 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:54.438 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:54.438 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:54.438 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:54.438 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:54.439 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:54.439 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:54.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:54.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:54.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:54.544 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:54.545 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:54.545 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:54.545 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:54.545 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:54.545 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:54.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:38:54.911 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:54.911 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:54.912 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:55.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:38:55.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:38:55.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:55.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:55.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:55.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:55.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:55.456 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:55.456 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:56.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19220234 2023-11-02T09:38:56.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19220234 '!=' null 2023-11-02T09:38:56.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:56.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:56.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:56.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:56.094 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:56.094 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:56.094 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:56.095 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:56.095 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:56.095 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:56.095 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:56.096 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:56.096 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:56.096 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:56.096 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:56.096 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:56.097 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:56.097 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:56.097 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:56.097 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:56.097 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:56.098 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:56.098 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:56.098 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:56.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:56.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:56.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:56.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:56.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:56.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:56.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:56.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:56.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:56.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:38:56.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:56.570 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:56.570 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:57.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:38:57.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:38:57.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:57.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:57.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:57.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:57.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:57.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:57.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:57.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19220259 2023-11-02T09:38:57.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19220259 '!=' null 2023-11-02T09:38:57.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:57.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:57.825 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:57.825 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:57.901 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:57.902 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:57.902 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:57.902 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:57.902 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:57.903 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:57.903 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:57.903 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:57.903 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:57.904 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:57.904 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:57.904 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:57.904 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:57.904 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:57.905 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:57.905 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:57.905 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:57.905 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:57.905 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:57.906 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:57.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:58.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:58.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:58.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:58.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:58.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:58.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:58.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:58.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:38:58.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:38:58.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:38:58.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:38:58.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:38:58.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:38:58.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:38:58.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:38:58.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:38:58.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:38:58.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:38:58.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:38:58.884 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:38:58.884 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:38:59.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27956752 2023-11-02T09:38:59.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27956752 '!=' null 2023-11-02T09:38:59.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:38:59.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:59.528 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:38:59.529 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:38:59.604 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:38:59.605 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:38:59.605 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:59.606 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:59.606 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:59.606 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:38:59.606 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:38:59.607 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:38:59.607 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:38:59.607 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:38:59.607 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:59.607 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:38:59.608 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:38:59.608 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:38:59.608 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:38:59.608 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:38:59.608 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:38:59.609 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:38:59.609 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:38:59.609 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:38:59.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:38:59.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:38:59.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:38:59.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:38:59.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:38:59.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:38:59.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:38:59.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:38:59.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:00.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:00.167 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:00.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:00.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:00.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:00.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:00.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:00.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:00.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:00.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:00.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:00.825 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:00.825 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:01.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18346703 2023-11-02T09:39:01.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18346703 '!=' null 2023-11-02T09:39:01.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:01.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:01.360 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:01.360 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:01.437 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:01.437 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:01.437 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:01.438 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:01.438 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:01.438 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:01.438 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:01.438 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:01.439 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:01.439 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:01.439 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:01.439 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:01.439 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:01.439 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:01.440 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:01.440 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:01.440 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:01.440 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:01.440 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:01.441 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:01.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:01.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:01.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:01.548 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:01.548 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:01.548 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:01.548 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:01.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:01.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:01.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:01.984 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:01.984 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:01.984 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:02.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:02.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:02.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:02.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:02.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:02.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:02.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:02.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:02.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:03.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27083615 2023-11-02T09:39:03.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27083615 '!=' null 2023-11-02T09:39:03.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:03.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:03.224 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:03.224 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:03.301 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:03.301 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:03.301 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:03.301 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:03.301 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:03.302 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:03.302 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:03.302 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:03.302 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:03.302 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:03.303 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:03.303 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:03.303 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:03.303 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:03.303 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:03.304 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:03.304 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:03.304 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:03.304 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:03.304 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:03.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:03.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:03.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:03.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:03.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:03.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:03.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:03.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:03.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:03.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:03.781 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:03.781 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:03.781 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:04.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:04.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:04.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:04.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:04.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:04.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:04.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:04.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:04.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:04.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473439 2023-11-02T09:39:04.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473439 '!=' null 2023-11-02T09:39:04.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:04.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:04.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:04.964 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:05.041 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:05.041 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:05.041 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:05.042 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:05.042 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:05.042 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:05.042 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:05.043 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:05.043 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:05.043 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:05.043 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:05.043 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:05.044 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:05.044 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:05.044 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:05.044 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:05.044 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:05.045 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:05.045 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:05.045 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:05.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:05.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:05.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:05.150 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:05.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:05.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:05.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:05.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:05.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:05.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:05.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:05.517 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:05.517 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:06.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:06.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:06.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:06.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:06.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:06.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:06.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:06.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:06.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:06.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473204 2023-11-02T09:39:06.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473204 '!=' null 2023-11-02T09:39:06.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:06.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:06.896 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:06.896 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:06.972 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:06.972 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:06.973 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:06.973 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:06.973 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:06.973 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:06.973 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:06.974 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:06.974 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:06.974 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:06.974 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:06.974 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:06.975 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:06.975 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:06.975 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:06.975 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:06.975 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:06.976 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:06.976 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:06.976 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:06.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:07.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:07.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:07.082 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:07.082 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:07.083 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:07.083 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:07.083 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:07.083 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:07.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:07.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:07.451 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:07.451 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:08.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:08.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:08.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:08.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:08.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:08.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:08.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:08.005 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:08.005 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:08.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473204 2023-11-02T09:39:08.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473204 '!=' null 2023-11-02T09:39:08.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:08.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:08.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:08.693 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:08.769 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:08.770 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:08.770 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:08.770 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:08.770 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:08.770 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:08.771 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:08.771 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:08.771 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:08.771 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:08.771 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:08.772 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:08.772 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:08.772 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:08.772 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:08.773 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:08.773 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:08.773 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:08.773 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:08.773 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:08.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:08.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:08.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:08.879 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:08.879 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:08.880 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:08.880 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:08.880 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:08.880 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:09.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:09.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:09.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:09.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:09.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:09.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:09.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:09.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:09.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:09.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:09.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:09.912 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:09.913 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:10.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27083467 2023-11-02T09:39:10.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27083467 '!=' null 2023-11-02T09:39:10.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:10.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:10.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:10.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:10.554 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:10.555 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:10.555 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:10.555 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:10.555 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:10.556 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:10.556 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:10.556 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:10.556 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:10.556 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:10.557 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:10.557 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:10.557 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:10.557 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:10.557 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:10.557 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:10.558 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:10.558 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:10.558 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:10.558 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:10.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:10.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:10.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:10.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:10.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:10.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:10.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:10.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:10.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:11.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:11.032 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:11.032 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:11.032 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:11.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:11.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:11.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:11.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:11.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:11.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:11.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:11.622 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:11.622 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:12.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18346926 2023-11-02T09:39:12.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18346926 '!=' null 2023-11-02T09:39:12.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:12.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:12.160 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:12.160 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:12.238 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:12.238 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:12.238 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:12.238 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:12.239 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:12.239 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:12.239 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:12.239 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:12.239 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:12.240 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:12.240 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:12.240 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:12.240 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:12.240 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:12.241 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:12.241 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:12.241 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:12.241 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:12.241 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:12.241 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:12.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:12.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:12.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:12.345 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:12.346 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:12.346 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:12.346 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:12.346 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:12.346 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:12.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:12.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:12.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:12.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:13.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:13.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:13.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:13.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:13.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:13.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:13.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:13.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:13.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:13.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209887 2023-11-02T09:39:13.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209887 '!=' null 2023-11-02T09:39:13.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:13.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:13.838 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:13.838 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:13.913 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:13.914 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:13.914 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:13.914 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:13.914 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:13.915 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:13.915 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:13.915 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:13.915 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:13.915 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:13.915 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:13.916 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:13.916 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:13.916 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:13.916 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:13.916 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:13.917 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:13.917 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:13.917 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:13.918 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:13.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:14.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:14.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:14.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:14.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:14.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:14.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:14.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:14.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:14.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:14.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:14.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:14.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:15.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:15.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:15.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:15.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:15.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:15.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:15.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:15.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:15.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:15.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473211 2023-11-02T09:39:15.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473211 '!=' null 2023-11-02T09:39:15.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:15.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:15.761 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:15.762 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:15.839 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:15.839 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:15.839 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:15.839 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:15.840 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:15.840 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:15.840 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:15.840 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:15.840 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:15.841 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:15.841 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:15.841 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:15.841 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:15.841 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:15.842 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:15.842 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:15.842 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:15.842 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:15.842 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:15.843 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:15.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:15.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:15.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:15.947 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:15.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:15.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:15.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:15.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:15.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:16.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:16.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:16.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:16.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:16.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:16.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:16.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:16.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:16.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:16.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:16.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:16.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:16.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:17.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473185 2023-11-02T09:39:17.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473185 '!=' null 2023-11-02T09:39:17.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:17.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:17.368 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:17.368 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:17.444 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:17.445 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:17.445 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:17.445 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:17.445 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:17.445 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:17.446 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:17.446 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:17.446 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:17.446 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:17.447 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:17.447 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:17.447 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:17.447 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:17.447 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:17.448 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:17.448 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:17.448 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:17.448 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:17.448 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:17.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:17.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:17.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:17.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:17.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:17.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:17.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:17.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:17.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:17.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:17.921 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:17.921 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:17.922 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:18.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:18.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:18.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:18.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:18.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:18.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:18.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:18.471 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:18.471 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:19.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210150 2023-11-02T09:39:19.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210150 '!=' null 2023-11-02T09:39:19.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:19.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:19.014 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:19.014 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:19.091 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:19.091 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:19.091 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:19.091 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:19.092 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:19.092 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:19.092 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:19.092 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:19.092 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:19.093 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:19.093 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:19.093 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:19.093 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:19.093 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:19.093 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:19.094 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:19.094 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:19.094 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:19.094 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:19.094 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:19.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:19.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:19.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:19.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:19.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:19.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:19.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:19.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:19.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:19.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:19.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:19.573 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:19.573 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:20.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:20.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:20.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:20.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:20.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:20.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:20.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:20.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:20.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:20.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599766 2023-11-02T09:39:20.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599766 '!=' null 2023-11-02T09:39:20.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:20.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:20.869 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:20.869 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:20.945 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:20.946 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:20.946 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:20.946 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:20.946 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:20.947 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:20.947 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:20.947 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:20.947 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:20.947 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:20.948 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:20.948 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:20.948 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:20.948 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:20.948 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:20.949 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:20.949 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:20.949 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:20.949 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:20.949 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:20.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:21.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:21.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:21.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:21.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:21.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:21.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:21.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:21.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:21.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:21.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:21.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:21.422 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:22.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:22.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:22.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:22.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:22.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:22.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:22.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:22.026 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:22.026 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:22.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336545 2023-11-02T09:39:22.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336545 '!=' null 2023-11-02T09:39:22.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:22.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:22.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:22.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:22.794 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:22.794 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:22.794 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:22.795 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:22.795 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:22.795 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:22.795 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:22.795 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:22.795 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:22.796 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:22.796 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:22.796 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:22.796 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:22.796 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:22.797 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:22.797 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:22.797 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:22.797 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:22.797 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:22.798 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:22.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:22.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:22.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:22.904 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:22.904 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:22.904 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:22.904 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:22.904 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:22.905 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:23.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:23.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:23.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:23.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:23.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:23.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:23.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:23.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:23.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:23.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:23.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:23.836 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:23.836 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:24.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336545 2023-11-02T09:39:24.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336545 '!=' null 2023-11-02T09:39:24.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:24.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:24.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:24.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:24.459 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:24.459 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:24.460 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:24.460 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:24.460 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:24.460 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:24.460 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:24.461 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:24.461 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:24.461 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:24.461 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:24.461 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:24.462 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:24.462 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:24.462 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:24.462 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:24.462 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:24.463 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:24.463 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:24.463 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:24.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:24.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:24.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:24.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:24.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:24.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:24.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:24.570 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:24.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:24.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:24.975 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:24.976 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:24.976 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:25.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:25.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:25.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:25.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:25.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:25.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:25.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:25.541 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:25.541 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:26.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473513 2023-11-02T09:39:26.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473513 '!=' null 2023-11-02T09:39:26.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:26.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:26.116 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:26.116 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:26.191 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:26.192 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:26.192 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:26.192 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:26.192 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:26.193 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:26.193 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:26.193 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:26.193 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:26.193 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:26.194 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:26.194 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:26.194 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:26.194 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:26.194 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:26.195 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:26.195 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:26.195 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:26.195 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:26.195 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:26.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:26.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:26.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:26.301 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:26.301 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:26.301 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:26.302 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:26.302 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:26.302 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:26.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:26.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:26.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:26.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:27.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:27.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:27.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:27.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:27.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:27.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:27.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:27.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:27.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:27.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473531 2023-11-02T09:39:27.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473531 '!=' null 2023-11-02T09:39:27.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:27.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:27.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:27.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:28.033 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:28.034 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:28.034 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:28.034 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:28.034 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:28.034 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:28.035 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:28.035 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:28.035 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:28.035 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:28.035 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:28.035 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:28.036 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:28.036 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:28.036 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:28.037 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:28.037 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:28.038 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:28.038 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:28.038 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:28.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:28.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:28.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:28.143 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:28.143 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:28.143 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:28.143 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:28.143 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:28.144 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:28.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:28.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:28.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:28.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:29.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:29.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:29.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:29.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:29.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:29.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:29.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:29.036 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:29.037 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:29.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209968 2023-11-02T09:39:29.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209968 '!=' null 2023-11-02T09:39:29.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:29.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:29.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:29.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:29.822 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:29.823 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:29.823 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:29.823 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:29.823 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:29.823 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:29.824 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:29.824 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:29.824 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:29.824 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:29.824 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:29.825 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:29.825 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:29.825 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:29.825 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:29.825 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:29.825 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:29.826 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:29.826 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:29.832 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:29.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:29.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:29.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:29.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:29.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:29.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:29.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:29.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:29.933 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:30.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:30.296 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:30.296 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:30.296 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:30.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:30.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:30.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:30.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:30.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:30.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:30.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:30.859 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:30.859 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:31.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473297 2023-11-02T09:39:31.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473297 '!=' null 2023-11-02T09:39:31.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:31.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:31.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:31.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:31.471 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:31.472 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:31.472 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:31.472 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:31.472 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:31.473 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:31.473 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:31.473 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:31.473 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:31.473 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:31.474 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:31.474 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:31.474 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:31.474 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:31.474 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:31.474 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:31.475 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:31.475 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:31.475 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:31.475 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:31.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:31.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:31.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:31.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:31.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:31.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:31.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:31.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:31.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:31.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:31.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:31.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:31.950 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:32.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:32.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:32.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:32.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:32.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:32.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:32.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:32.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:32.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:33.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209899 2023-11-02T09:39:33.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209899 '!=' null 2023-11-02T09:39:33.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:33.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:33.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:33.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:33.161 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:33.162 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:33.162 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:33.162 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:33.162 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:33.163 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:33.163 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:33.163 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:33.163 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:33.163 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:33.164 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:33.164 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:33.164 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:33.164 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:33.164 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:33.165 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:33.165 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:33.165 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:33.165 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:33.165 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:33.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:33.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:33.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:33.272 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:33.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:33.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:33.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:33.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:33.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:33.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:33.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:33.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:33.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:34.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:34.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:34.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:34.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:34.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:34.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:34.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:34.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:34.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:34.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473122 2023-11-02T09:39:34.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473122 '!=' null 2023-11-02T09:39:34.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:34.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:34.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:34.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:34.967 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:34.967 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:34.967 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:34.968 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:34.968 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:34.968 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:34.968 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:34.968 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:34.969 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:34.969 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:34.969 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:34.969 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:34.969 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:34.970 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:34.970 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:34.970 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:34.970 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:34.970 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:34.970 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:34.971 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:34.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:35.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:35.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:35.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:35.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:35.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:35.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:35.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:35.079 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:35.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:35.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:35.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:35.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:35.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:35.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:35.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:35.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:35.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:35.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:35.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:35.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:35.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:36.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473122 2023-11-02T09:39:36.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473122 '!=' null 2023-11-02T09:39:36.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:36.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:36.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:36.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:36.739 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:36.739 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:36.740 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:36.740 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:36.740 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:36.740 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:36.740 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:36.741 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:36.741 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:36.741 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:36.741 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:36.741 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:36.742 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:36.742 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:36.742 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:36.742 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:36.742 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:36.742 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:36.743 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:36.743 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:36.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:36.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:36.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:36.849 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:36.849 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:36.849 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:36.849 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:36.849 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:36.850 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:37.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:37.215 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:37.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:37.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:37.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:37.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:37.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:37.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:37.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:37.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:37.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:37.761 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:37.761 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:38.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473095 2023-11-02T09:39:38.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473095 '!=' null 2023-11-02T09:39:38.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:38.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:38.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:38.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:38.342 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:38.342 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:38.343 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:38.343 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:38.343 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:38.343 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:38.343 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:38.344 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:38.344 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:38.344 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:38.344 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:38.344 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:38.345 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:38.345 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:38.345 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:38.345 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:38.345 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:38.346 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:38.346 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:38.346 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:38.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:38.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:38.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:38.452 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:38.452 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:38.452 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:38.452 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:38.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:38.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:39.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:39.030 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:39.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:39.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:39.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:39.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:39.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:39.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:39.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:39.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:39.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:39.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:39.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:40.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209538 2023-11-02T09:39:40.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209538 '!=' null 2023-11-02T09:39:40.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:40.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:40.185 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:40.185 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:40.263 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:40.264 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:40.264 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:40.264 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:40.264 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:40.264 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:40.265 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:40.265 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:40.265 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:40.265 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:40.266 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:40.266 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:40.266 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:40.266 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:40.266 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:40.267 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:40.267 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:40.267 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:40.267 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:40.267 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:40.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:40.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:40.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:40.373 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:40.373 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:40.374 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:40.374 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:40.374 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:40.374 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:40.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:40.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:40.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:40.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:41.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:41.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:41.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:41.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:41.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:41.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:41.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:41.350 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:41.350 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:42.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473192 2023-11-02T09:39:42.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473192 '!=' null 2023-11-02T09:39:42.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:42.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:42.001 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:42.001 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:42.079 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:42.079 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:42.079 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:42.079 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:42.080 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:42.080 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:42.080 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:42.080 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:42.080 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:42.081 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:42.083 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:42.083 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:42.084 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:42.084 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:42.084 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:42.084 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:42.084 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:42.084 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:42.085 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:42.085 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:42.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:42.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:42.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:42.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:42.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:42.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:42.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:42.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:42.189 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:42.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:42.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:42.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:42.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:43.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:43.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:43.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:43.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:43.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:43.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:43.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:43.102 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:43.102 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:43.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209799 2023-11-02T09:39:43.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209799 '!=' null 2023-11-02T09:39:43.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:43.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:43.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:43.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:43.739 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:43.739 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:43.740 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:43.740 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:43.740 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:43.740 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:43.740 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:43.741 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:43.741 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:43.741 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:43.741 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:43.741 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:43.741 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:43.742 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:43.742 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:43.742 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:43.742 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:43.742 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:43.743 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:43.743 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:43.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:43.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:43.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:43.850 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:43.851 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:43.851 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:43.851 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:43.851 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:43.851 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:44.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:44.253 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:44.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:44.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:44.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:44.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:44.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:44.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:44.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:44.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:44.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:44.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:44.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:45.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473267 2023-11-02T09:39:45.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473267 '!=' null 2023-11-02T09:39:45.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:45.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:45.405 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:45.405 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:45.480 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:45.481 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:45.481 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:45.481 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:45.481 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:45.482 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:45.482 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:45.482 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:45.482 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:45.482 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:45.482 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:45.483 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:45.483 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:45.483 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:45.483 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:45.483 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:45.484 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:45.484 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:45.484 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:45.485 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:45.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:45.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:45.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:45.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:45.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:45.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:45.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:45.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:45.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:45.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:45.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:45.960 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:45.960 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:46.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:46.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:46.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:46.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:46.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:46.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:46.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:46.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:46.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:47.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473304 2023-11-02T09:39:47.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473304 '!=' null 2023-11-02T09:39:47.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:47.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:47.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:47.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:47.128 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:47.128 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:47.128 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:47.128 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:47.129 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:47.129 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:47.129 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:47.129 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:47.129 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:47.130 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:47.130 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:47.130 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:47.130 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:47.130 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:47.131 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:47.131 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:47.131 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:47.131 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:47.131 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:47.131 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:47.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:47.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:47.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:47.237 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:47.237 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:47.237 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:47.238 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:47.238 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:47.238 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:47.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:47.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:47.609 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:47.609 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:48.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:48.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:48.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:48.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:48.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:48.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:48.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:48.172 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:48.173 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:49.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209792 2023-11-02T09:39:49.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209792 '!=' null 2023-11-02T09:39:49.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:49.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:49.063 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:49.063 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:49.139 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:49.140 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:49.140 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:49.140 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:49.140 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:49.141 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:49.141 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:49.141 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:49.141 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:49.141 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:49.142 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:49.142 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:49.142 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:49.142 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:49.142 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:49.143 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:49.143 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:49.143 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:49.143 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:49.143 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:49.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:49.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:49.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:49.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:49.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:49.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:49.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:49.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:49.250 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:49.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:49.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:49.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:49.616 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:50.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:50.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:50.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:50.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:50.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:50.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:50.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:50.206 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:50.206 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:50.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473012 2023-11-02T09:39:50.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473012 '!=' null 2023-11-02T09:39:50.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:50.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:50.947 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:50.947 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:51.022 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:51.022 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:51.023 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:51.023 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:51.023 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:51.023 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:51.023 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:51.024 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:51.024 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:51.024 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:51.024 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:51.024 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:51.024 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:51.025 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:51.025 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:51.025 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:51.025 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:51.025 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:51.026 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:51.026 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:51.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:51.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:51.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:51.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:51.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:51.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:51.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:51.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:51.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:51.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:51.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:51.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:51.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:52.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:52.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:52.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:52.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:52.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:52.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:52.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:52.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:52.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:52.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473012 2023-11-02T09:39:52.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473012 '!=' null 2023-11-02T09:39:52.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:52.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:52.589 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:52.589 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:52.665 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:52.665 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:52.665 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:52.665 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:52.666 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:52.666 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:52.666 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:52.666 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:52.666 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:52.667 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:52.667 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:52.667 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:52.667 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:52.667 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:52.668 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:52.668 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:52.668 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:52.668 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:52.669 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:52.669 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:52.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:52.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:52.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:52.774 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:52.774 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:52.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:52.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:52.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:52.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:53.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:53.142 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:53.142 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:53.142 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:53.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:53.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:53.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:53.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:53.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:53.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:53.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:53.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:53.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:54.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27082978 2023-11-02T09:39:54.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27082978 '!=' null 2023-11-02T09:39:54.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:54.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:54.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:54.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:54.362 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:54.362 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:54.362 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:54.362 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:54.363 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:54.363 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:54.363 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:54.363 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:54.363 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:54.364 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:54.364 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:54.364 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:54.364 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:54.364 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:54.364 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:54.365 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:54.365 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:54.365 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:54.365 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:54.365 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:54.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:54.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:54.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:54.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:54.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:54.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:54.473 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:54.473 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:54.473 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:54.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:54.938 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:54.938 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:54.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:55.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:55.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:55.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:55.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:55.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:55.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:55.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:55.447 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:55.447 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:56.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18346509 2023-11-02T09:39:56.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18346509 '!=' null 2023-11-02T09:39:56.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:56.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:56.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:56.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:56.100 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:56.100 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:56.101 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:56.101 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:56.101 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:56.101 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:56.101 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:56.102 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:56.102 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:56.102 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:56.102 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:56.102 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:56.103 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:56.103 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:56.103 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:56.103 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:56.104 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:56.104 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:56.104 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:56.104 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:56.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:56.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:56.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:56.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:56.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:56.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:56.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:56.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:56.211 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:56.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:56.691 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:56.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:56.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:57.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:57.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:57.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:57.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:57.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:57.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:57.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:57.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:57.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:57.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18346477 2023-11-02T09:39:57.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18346477 '!=' null 2023-11-02T09:39:57.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:57.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:57.825 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:57.825 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:57.901 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:57.901 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:57.901 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:57.901 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:57.901 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:57.902 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:57.902 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:57.902 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:57.902 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:57.903 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:57.903 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:57.903 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:57.903 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:57.903 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:57.904 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:57.904 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:57.904 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:57.904 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:57.904 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:57.904 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:57.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:58.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:58.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:58.010 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:58.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:58.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:58.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:58.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:58.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:39:58.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:39:58.381 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:39:58.382 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:39:58.382 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:39:58.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:39:58.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:39:58.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:39:58.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:39:58.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:39:58.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:39:58.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:39:58.893 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:39:58.893 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:39:59.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27082935 2023-11-02T09:39:59.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27082935 '!=' null 2023-11-02T09:39:59.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:39:59.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:59.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:39:59.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:39:59.659 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:39:59.659 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:39:59.660 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:59.660 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:59.660 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:59.660 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:39:59.660 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:39:59.661 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:39:59.661 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:39:59.661 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:39:59.661 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:59.661 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:39:59.668 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:39:59.669 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:39:59.669 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:39:59.669 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:39:59.669 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:39:59.669 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:39:59.670 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:39:59.670 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:39:59.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:39:59.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:39:59.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:39:59.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:39:59.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:39:59.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:39:59.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:39:59.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:39:59.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:40:00.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:40:00.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:40:00.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:40:00.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:40:00.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:40:00.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:40:00.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:40:00.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:40:00.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:40:00.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:40:00.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:40:00.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:40:00.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:40:01.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18346587 2023-11-02T09:40:01.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18346587 '!=' null 2023-11-02T09:40:01.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:40:01.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:01.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:40:01.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:01.267 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:40:01.267 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:40:01.268 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:01.268 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:01.268 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:01.268 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:40:01.268 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:01.269 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:40:01.269 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:40:01.269 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:40:01.269 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:01.269 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:01.270 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:01.270 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:40:01.270 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:01.270 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:01.271 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:01.271 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:40:01.271 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:01.271 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:40:01.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:40:01.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:40:01.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:40:01.379 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:40:01.379 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:40:01.379 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:40:01.379 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:40:01.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:40:01.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:40:01.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:40:01.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:40:01.792 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:40:01.792 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:40:02.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:40:02.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:40:02.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:40:02.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:40:02.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:40:02.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:40:02.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:40:02.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:40:02.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:40:02.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209723 2023-11-02T09:40:02.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209723 '!=' null 2023-11-02T09:40:02.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:40:02.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:02.910 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:40:02.910 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:02.986 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:40:02.987 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:40:02.987 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:02.987 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:02.987 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:02.987 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:40:02.988 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:02.988 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:40:02.988 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:40:02.988 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:40:02.988 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:02.988 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:02.989 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:02.989 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:40:02.989 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:02.989 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:02.989 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:02.990 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:40:02.990 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:02.990 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:40:02.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:40:03.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:40:03.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:40:03.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:40:03.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:40:03.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:40:03.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:40:03.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:40:03.097 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:40:03.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:40:03.519 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:40:03.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:40:03.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:40:04.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:40:04.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:40:04.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:40:04.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:40:04.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:40:04.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:40:04.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:40:04.026 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:40:04.026 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:40:04.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209723 2023-11-02T09:40:04.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209723 '!=' null 2023-11-02T09:40:04.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:40:04.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:04.575 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:40:04.575 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:04.651 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:40:04.651 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:40:04.651 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:04.652 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:04.652 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:04.652 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:40:04.652 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:04.652 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:40:04.653 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:40:04.653 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:40:04.653 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:04.653 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:04.653 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:04.654 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:40:04.654 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:04.654 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:04.654 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:04.654 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:40:04.654 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:04.655 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:40:04.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:40:04.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:40:04.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:40:04.765 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:40:04.765 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:40:04.766 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:40:04.766 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:40:04.766 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:40:04.766 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:40:05.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:40:05.184 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:40:05.185 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:40:05.185 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:40:05.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:40:05.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:40:05.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:40:05.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:40:05.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:40:05.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:40:05.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:40:05.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:40:05.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:40:06.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599817 2023-11-02T09:40:06.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599817 '!=' null 2023-11-02T09:40:06.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:40:06.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:06.272 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:40:06.272 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:06.348 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:40:06.348 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:40:06.349 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:06.349 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:06.349 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:06.349 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:40:06.349 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:06.350 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:40:06.350 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:40:06.350 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:40:06.350 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:06.350 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:06.351 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:06.351 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:40:06.351 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:06.351 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:06.351 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:06.351 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:40:06.352 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:06.352 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:40:06.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:40:06.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:40:06.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:40:06.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:40:06.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:40:06.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:40:06.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:40:06.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:40:06.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:40:07.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:40:07.014 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:40:07.015 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:40:07.015 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:40:07.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:40:07.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:40:07.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:40:07.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:40:07.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:40:07.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:40:07.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:40:07.528 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:40:07.529 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:40:08.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599837 2023-11-02T09:40:08.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599837 '!=' null 2023-11-02T09:40:08.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:40:08.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:08.070 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:40:08.070 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:08.146 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:40:08.147 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:40:08.147 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:08.147 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:08.147 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:08.148 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:40:08.148 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:08.148 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:40:08.148 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:40:08.148 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:40:08.149 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:08.149 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:08.149 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:08.149 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:40:08.149 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:08.150 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:08.150 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:08.150 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:40:08.150 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:08.150 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:40:08.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:40:08.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:40:08.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:40:08.256 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:40:08.256 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:40:08.256 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:40:08.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:40:08.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:40:08.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:40:08.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:40:08.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:40:08.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:40:08.629 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:40:09.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:40:09.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:40:09.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:40:09.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:40:09.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:40:09.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:40:09.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:40:09.167 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:40:09.167 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:40:09.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336561 2023-11-02T09:40:09.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336561 '!=' null 2023-11-02T09:40:09.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:40:09.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:09.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:40:09.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:10.002 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:40:10.002 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:40:10.003 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:10.003 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:10.003 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:10.003 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:40:10.003 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:10.004 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:40:10.004 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:40:10.004 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:40:10.004 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:10.004 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:10.005 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:10.005 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:40:10.005 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:10.005 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:10.005 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:10.006 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:40:10.006 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:10.006 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:40:10.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:40:10.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:40:10.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:40:10.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:40:10.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:40:10.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:40:10.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:40:10.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:40:10.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:40:10.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:40:10.476 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:40:10.476 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:40:10.477 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:40:11.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:40:11.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:40:11.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:40:11.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:40:11.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:40:11.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:40:11.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:40:11.067 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:40:11.067 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:40:11.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599828 2023-11-02T09:40:11.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599828 '!=' null 2023-11-02T09:40:11.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:40:11.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:11.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:40:11.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:11.688 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:40:11.688 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:40:11.689 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:11.689 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:11.689 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:11.689 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:40:11.689 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:11.690 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:40:11.690 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:40:11.690 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:40:11.690 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:11.690 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:11.691 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:11.691 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:40:11.691 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:11.691 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:11.691 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:11.691 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:40:11.692 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:11.692 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:40:11.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:40:11.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:40:11.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:40:11.797 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:40:11.797 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:40:11.798 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:40:11.798 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:40:11.798 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:40:11.798 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:40:12.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:40:12.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:40:12.183 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:40:12.183 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:40:12.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:40:12.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:40:12.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:40:12.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:40:12.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:40:12.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:40:12.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:40:12.693 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:40:12.694 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:40:13.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336623 2023-11-02T09:40:13.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336623 '!=' null 2023-11-02T09:40:13.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:40:13.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:13.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:40:13.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:13.424 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:40:13.424 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:40:13.424 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:13.425 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:13.425 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:13.425 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:40:13.425 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:13.425 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:40:13.426 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:40:13.426 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:40:13.426 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:13.426 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:13.426 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:13.427 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:40:13.427 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:13.427 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:13.427 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:13.427 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:40:13.428 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:13.428 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:40:13.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:40:13.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:40:13.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:40:13.533 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:40:13.533 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:40:13.534 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:40:13.534 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:40:13.534 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:40:13.534 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:40:13.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:40:13.902 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:40:13.903 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:40:13.903 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:40:14.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:40:14.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:40:14.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:40:14.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:40:14.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:40:14.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:40:14.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:40:14.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:40:14.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:40:14.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473641 2023-11-02T09:40:14.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473641 '!=' null 2023-11-02T09:40:14.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:40:14.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:14.995 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:40:14.996 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:15.071 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:40:15.072 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:40:15.072 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:15.072 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:15.072 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:15.072 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:40:15.073 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:15.073 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:40:15.073 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:40:15.073 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:40:15.073 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:15.074 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:15.074 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:15.074 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:40:15.074 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:15.074 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:15.075 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:15.075 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:40:15.075 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:15.075 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:40:15.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:40:15.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:40:15.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:40:15.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:40:15.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:40:15.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:40:15.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:40:15.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:40:15.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:40:15.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:40:15.545 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:40:15.546 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:40:15.546 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:40:16.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:40:16.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:40:16.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:40:16.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:40:16.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:40:16.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:40:16.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:40:16.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:40:16.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:40:16.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473641 2023-11-02T09:40:16.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473641 '!=' null 2023-11-02T09:40:16.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:40:16.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:16.703 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:40:16.703 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:16.780 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:40:16.780 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:40:16.781 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:16.781 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:16.781 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:16.781 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:40:16.781 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:16.782 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:40:16.782 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:40:16.782 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:40:16.782 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:16.782 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:16.783 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:16.783 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:40:16.783 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:16.783 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:16.783 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:16.784 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:40:16.784 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:16.784 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:40:16.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:40:16.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:40:16.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:40:16.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:40:16.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:40:16.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:40:16.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:40:16.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:40:16.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:40:17.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:40:17.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:40:17.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:40:17.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:40:17.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:40:17.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:40:17.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:40:17.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:40:17.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:40:17.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:40:17.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:40:17.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:40:17.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:40:18.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473457 2023-11-02T09:40:18.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473457 '!=' null 2023-11-02T09:40:18.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:40:18.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:18.500 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:40:18.500 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:18.576 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:40:18.577 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:40:18.577 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:18.577 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:18.577 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:18.578 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [ 2023-11-02T09:40:18.578 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:18.578 INFO:tasks.workunit.client.0.smithi116.stdout: "item": "3.0", 2023-11-02T09:40:18.578 INFO:tasks.workunit.client.0.smithi116.stdout: "prio": 190, 2023-11-02T09:40:18.578 INFO:tasks.workunit.client.0.smithi116.stdout: "can_preempt": true 2023-11-02T09:40:18.579 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:18.579 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:18.579 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:18.579 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:40:18.579 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:18.580 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:18.580 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:18.580 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:40:18.580 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:18.580 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:40:18.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:40:18.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:40:18.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:40:18.686 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:40:18.686 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:40:18.686 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:40:18.687 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:40:18.687 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:40:18.687 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:40:19.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:40:19.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:40:19.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:40:19.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:40:19.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:40:19.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:40:19.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:40:19.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:40:19.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:40:19.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:40:19.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:40:19.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:40:19.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:40:20.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209994 2023-11-02T09:40:20.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209994 '!=' null 2023-11-02T09:40:20.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:40:20.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:20.357 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:40:20.358 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:20.434 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:40:20.434 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:40:20.434 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:20.435 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:20.435 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:20.435 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:40:20.435 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:20.435 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:40:20.436 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:20.436 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:20.436 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:20.436 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:40:20.436 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:20.437 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:40:20.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:40:20.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:40:20.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:40:20.544 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:40:20.545 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:40:20.545 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:40:20.545 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:40:20.546 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:40:20.546 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:40:20.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2023-11-02T09:40:20.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:40:20.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:40:20.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:40:21.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2023-11-02T09:40:21.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2023-11-02T09:40:21.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-11-02T09:40:21.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-11-02T09:40:21.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-11-02T09:40:21.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-11-02T09:40:21.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-11-02T09:40:21.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-11-02T09:40:21.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-11-02T09:40:22.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473316 2023-11-02T09:40:22.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473316 '!=' null 2023-11-02T09:40:22.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2023-11-02T09:40:22.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:22.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2023-11-02T09:40:22.050 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_recovery_reservations 2023-11-02T09:40:22.125 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:40:22.125 INFO:tasks.workunit.client.0.smithi116.stdout: "local_reservations": { 2023-11-02T09:40:22.125 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:22.126 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:22.126 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:22.126 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:40:22.126 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:22.127 INFO:tasks.workunit.client.0.smithi116.stdout: "remote_reservations": { 2023-11-02T09:40:22.127 INFO:tasks.workunit.client.0.smithi116.stdout: "max_allowed": 1, 2023-11-02T09:40:22.127 INFO:tasks.workunit.client.0.smithi116.stdout: "min_priority": 0, 2023-11-02T09:40:22.127 INFO:tasks.workunit.client.0.smithi116.stdout: "queues": [], 2023-11-02T09:40:22.127 INFO:tasks.workunit.client.0.smithi116.stdout: "in_progress": [] 2023-11-02T09:40:22.128 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:22.128 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:40:22.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-11-02T09:40:22.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-11-02T09:40:22.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:40:22.234 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:40:22.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:40:22.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:40:22.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:40:22.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:40:22.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:40:22.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2023-11-02T09:40:22.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:40:22.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:40:22.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:40:23.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2023-11-02T09:40:23.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T09:40:23.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T09:40:23.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:298: TEST_recovery_priority: ceph pg dump pgs 2023-11-02T09:40:23.454 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T09:40:23.455 INFO:tasks.workunit.client.0.smithi116.stdout:7.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2023-11-02T09:37:01.608085+0000 91'600 117:1100 [1,4] 1 [1,4] 1 0'0 2023-11-02T09:29:12.218256+0000 0'0 2023-11-02T09:29:12.218256+0000 0 0 periodic scrub scheduled @ 2023-11-03T21:18:35.357196+0000 0 0 2023-11-02T09:40:23.455 INFO:tasks.workunit.client.0.smithi116.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2023-11-02T09:40:21.288784+0000 91'600 118:1109 [1,2] 1 [1,2] 1 0'0 2023-11-02T09:29:01.559968+0000 0'0 2023-11-02T09:29:01.559968+0000 0 0 periodic scrub scheduled @ 2023-11-03T21:04:10.893360+0000 0 0 2023-11-02T09:40:23.455 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2023-11-02T09:38:43.345444+0000 91'600 117:1113 [1,2] 1 [1,2] 1 0'0 2023-11-02T09:28:56.349957+0000 0'0 2023-11-02T09:28:56.349957+0000 0 0 periodic scrub scheduled @ 2023-11-03T15:26:34.392499+0000 0 0 2023-11-02T09:40:23.455 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T09:40:23.456 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T09:40:23.456 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T09:40:23.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: get_asok_path osd.1 2023-11-02T09:40:23.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-11-02T09:40:23.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-11-02T09:40:23.464 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:40:23.465 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:40:23.465 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:40:23.465 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.261297/ceph-osd.1.asok 2023-11-02T09:40:23.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: CEPH_ARGS= 2023-11-02T09:40:23.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.261297/ceph-osd.1.asok dump_pgstate_history 2023-11-02T09:40:23.543 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2023-11-02T09:40:23.543 INFO:tasks.workunit.client.0.smithi116.stdout: "pgs": [ 2023-11-02T09:40:23.544 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.544 INFO:tasks.workunit.client.0.smithi116.stdout: "pg": "7.0", 2023-11-02T09:40:23.544 INFO:tasks.workunit.client.0.smithi116.stdout: "currently": "Started/Primary/Active/Clean", 2023-11-02T09:40:23.544 INFO:tasks.workunit.client.0.smithi116.stdout: "history": [ 2023-11-02T09:40:23.544 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.545 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "60", 2023-11-02T09:40:23.545 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.545 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.545 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Start", 2023-11-02T09:40:23.545 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:13.009986+0000", 2023-11-02T09:40:23.546 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:13.009993+0000" 2023-11-02T09:40:23.546 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.546 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.548 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetInfo", 2023-11-02T09:40:23.548 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:13.009998+0000", 2023-11-02T09:40:23.548 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:13.107644+0000" 2023-11-02T09:40:23.548 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.549 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.549 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetLog", 2023-11-02T09:40:23.549 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:13.107645+0000", 2023-11-02T09:40:23.549 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:13.107697+0000" 2023-11-02T09:40:23.549 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.550 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.550 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetMissing", 2023-11-02T09:40:23.550 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:13.107697+0000", 2023-11-02T09:40:23.550 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:13.107706+0000" 2023-11-02T09:40:23.550 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.551 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.551 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2023-11-02T09:40:23.551 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:13.107706+0000", 2023-11-02T09:40:23.551 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:13.342113+0000" 2023-11-02T09:40:23.551 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.552 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.552 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering", 2023-11-02T09:40:23.552 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:13.009995+0000", 2023-11-02T09:40:23.552 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:13.342119+0000" 2023-11-02T09:40:23.552 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.553 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.553 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Activating", 2023-11-02T09:40:23.553 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:13.342285+0000", 2023-11-02T09:40:23.553 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:13.953091+0000" 2023-11-02T09:40:23.553 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.554 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.554 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Recovered", 2023-11-02T09:40:23.554 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:13.953092+0000", 2023-11-02T09:40:23.554 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:13.953148+0000" 2023-11-02T09:40:23.555 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.555 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.555 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Clean", 2023-11-02T09:40:23.555 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:13.953148+0000", 2023-11-02T09:40:23.555 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:14.388249+0000" 2023-11-02T09:40:23.556 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.556 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.556 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active", 2023-11-02T09:40:23.556 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:13.342120+0000", 2023-11-02T09:40:23.556 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:14.388267+0000" 2023-11-02T09:40:23.557 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.559 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.559 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary", 2023-11-02T09:40:23.559 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:13.009993+0000", 2023-11-02T09:40:23.559 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:14.388281+0000" 2023-11-02T09:40:23.560 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.560 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.560 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started", 2023-11-02T09:40:23.560 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:13.009983+0000", 2023-11-02T09:40:23.560 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:14.388284+0000" 2023-11-02T09:40:23.561 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.561 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.561 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.561 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.561 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "60", 2023-11-02T09:40:23.562 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.562 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.562 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Reset", 2023-11-02T09:40:23.562 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:14.388284+0000", 2023-11-02T09:40:23.562 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:14.388517+0000" 2023-11-02T09:40:23.563 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.563 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.563 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.563 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.563 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "89", 2023-11-02T09:40:23.564 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.564 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.567 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Start", 2023-11-02T09:40:23.567 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:14.388521+0000", 2023-11-02T09:40:23.574 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:14.388531+0000" 2023-11-02T09:40:23.574 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.575 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.575 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetInfo", 2023-11-02T09:40:23.575 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:14.388538+0000", 2023-11-02T09:40:23.575 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:14.506040+0000" 2023-11-02T09:40:23.575 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.580 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.581 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetLog", 2023-11-02T09:40:23.581 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:14.506041+0000", 2023-11-02T09:40:23.581 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:14.506096+0000" 2023-11-02T09:40:23.581 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.583 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.583 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetMissing", 2023-11-02T09:40:23.584 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:14.506096+0000", 2023-11-02T09:40:23.585 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:14.506106+0000" 2023-11-02T09:40:23.585 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.585 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.585 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2023-11-02T09:40:23.585 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:14.506106+0000", 2023-11-02T09:40:23.586 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:14.991613+0000" 2023-11-02T09:40:23.587 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.587 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.588 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering", 2023-11-02T09:40:23.588 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:14.388534+0000", 2023-11-02T09:40:23.588 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:14.991623+0000" 2023-11-02T09:40:23.588 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.589 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.589 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Activating", 2023-11-02T09:40:23.589 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:14.991734+0000", 2023-11-02T09:40:23.589 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:15.285792+0000" 2023-11-02T09:40:23.590 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.590 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.590 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Recovered", 2023-11-02T09:40:23.590 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:15.285792+0000", 2023-11-02T09:40:23.590 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:15.285824+0000" 2023-11-02T09:40:23.591 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.591 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.591 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Clean", 2023-11-02T09:40:23.591 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:15.285824+0000", 2023-11-02T09:40:23.592 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:43.036671+0000" 2023-11-02T09:40:23.592 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.592 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.592 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active", 2023-11-02T09:40:23.592 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:14.991624+0000", 2023-11-02T09:40:23.592 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:43.036698+0000" 2023-11-02T09:40:23.593 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.593 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.593 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary", 2023-11-02T09:40:23.593 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:14.388532+0000", 2023-11-02T09:40:23.594 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:43.036724+0000" 2023-11-02T09:40:23.594 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.594 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.594 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started", 2023-11-02T09:40:23.597 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:14.388518+0000", 2023-11-02T09:40:23.597 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:43.036729+0000" 2023-11-02T09:40:23.597 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.597 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.597 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.598 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.598 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "89", 2023-11-02T09:40:23.598 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.598 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.598 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Reset", 2023-11-02T09:40:23.599 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:43.036729+0000", 2023-11-02T09:40:23.599 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:43.037097+0000" 2023-11-02T09:40:23.599 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.599 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.599 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.600 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.600 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "96", 2023-11-02T09:40:23.600 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.600 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.600 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Start", 2023-11-02T09:40:23.601 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:43.037100+0000", 2023-11-02T09:40:23.601 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:43.037109+0000" 2023-11-02T09:40:23.601 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.601 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.602 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetInfo", 2023-11-02T09:40:23.602 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:43.037114+0000", 2023-11-02T09:40:23.602 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:43.037451+0000" 2023-11-02T09:40:23.602 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.602 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.604 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetLog", 2023-11-02T09:40:23.606 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:43.037452+0000", 2023-11-02T09:40:23.606 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:43.037499+0000" 2023-11-02T09:40:23.607 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.607 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.607 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetMissing", 2023-11-02T09:40:23.607 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:43.037499+0000", 2023-11-02T09:40:23.607 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:43.037510+0000" 2023-11-02T09:40:23.608 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.608 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.608 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2023-11-02T09:40:23.608 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:43.037511+0000", 2023-11-02T09:40:23.608 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:43.614758+0000" 2023-11-02T09:40:23.608 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.609 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.609 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering", 2023-11-02T09:40:23.609 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:43.037112+0000", 2023-11-02T09:40:23.609 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:43.614767+0000" 2023-11-02T09:40:23.609 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.610 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.610 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Activating", 2023-11-02T09:40:23.610 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:43.614826+0000", 2023-11-02T09:40:23.610 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:43.665218+0000" 2023-11-02T09:40:23.616 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.617 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.619 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Recovered", 2023-11-02T09:40:23.619 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:43.665219+0000", 2023-11-02T09:40:23.619 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:43.665247+0000" 2023-11-02T09:40:23.628 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.628 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.629 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Clean", 2023-11-02T09:40:23.629 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:43.665247+0000", 2023-11-02T09:40:23.629 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:29.325802+0000" 2023-11-02T09:40:23.630 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.631 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.631 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active", 2023-11-02T09:40:23.631 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:43.614768+0000", 2023-11-02T09:40:23.631 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:29.325819+0000" 2023-11-02T09:40:23.632 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.632 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.632 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary", 2023-11-02T09:40:23.633 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:43.037109+0000", 2023-11-02T09:40:23.633 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:29.325965+0000" 2023-11-02T09:40:23.633 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.633 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.634 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started", 2023-11-02T09:40:23.634 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:43.037097+0000", 2023-11-02T09:40:23.634 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:29.325968+0000" 2023-11-02T09:40:23.634 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.634 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.635 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.635 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.635 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "97", 2023-11-02T09:40:23.636 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.636 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.636 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Reset", 2023-11-02T09:40:23.636 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:29.325969+0000", 2023-11-02T09:40:23.637 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:29.326373+0000" 2023-11-02T09:40:23.637 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.637 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.637 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.638 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.639 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "99", 2023-11-02T09:40:23.639 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.639 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.640 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Start", 2023-11-02T09:40:23.640 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:29.326378+0000", 2023-11-02T09:40:23.640 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:29.326390+0000" 2023-11-02T09:40:23.641 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.649 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.649 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetInfo", 2023-11-02T09:40:23.660 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:29.326398+0000", 2023-11-02T09:40:23.661 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:29.445492+0000" 2023-11-02T09:40:23.661 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.661 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.661 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetLog", 2023-11-02T09:40:23.661 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:29.445493+0000", 2023-11-02T09:40:23.661 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:29.445546+0000" 2023-11-02T09:40:23.662 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.662 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.662 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering", 2023-11-02T09:40:23.662 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:29.326394+0000", 2023-11-02T09:40:23.662 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:29.445555+0000" 2023-11-02T09:40:23.663 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.663 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.663 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/WaitActingChange", 2023-11-02T09:40:23.663 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:29.445555+0000", 2023-11-02T09:40:23.663 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:29.890370+0000" 2023-11-02T09:40:23.664 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.664 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.664 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary", 2023-11-02T09:40:23.664 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:29.326390+0000", 2023-11-02T09:40:23.664 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:29.890412+0000" 2023-11-02T09:40:23.665 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.665 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.665 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started", 2023-11-02T09:40:23.665 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:29.326373+0000", 2023-11-02T09:40:23.665 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:29.890420+0000" 2023-11-02T09:40:23.665 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.666 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.666 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.666 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.666 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "99", 2023-11-02T09:40:23.666 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.667 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.667 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Reset", 2023-11-02T09:40:23.667 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:29.890421+0000", 2023-11-02T09:40:23.667 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:29.890763+0000" 2023-11-02T09:40:23.667 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.668 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.668 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.668 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.668 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "113", 2023-11-02T09:40:23.668 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.668 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.669 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Start", 2023-11-02T09:40:23.669 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:29.890769+0000", 2023-11-02T09:40:23.669 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:29.890781+0000" 2023-11-02T09:40:23.669 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.669 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.670 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetInfo", 2023-11-02T09:40:23.670 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:29.890790+0000", 2023-11-02T09:40:23.670 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:29.891335+0000" 2023-11-02T09:40:23.670 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.670 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.671 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetLog", 2023-11-02T09:40:23.671 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:29.891335+0000", 2023-11-02T09:40:23.671 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:29.891407+0000" 2023-11-02T09:40:23.671 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.671 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.671 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetMissing", 2023-11-02T09:40:23.672 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:29.891407+0000", 2023-11-02T09:40:23.672 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:29.891422+0000" 2023-11-02T09:40:23.672 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.672 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.672 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2023-11-02T09:40:23.673 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:29.891422+0000", 2023-11-02T09:40:23.673 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:30.309297+0000" 2023-11-02T09:40:23.673 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.673 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.673 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering", 2023-11-02T09:40:23.674 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:29.890785+0000", 2023-11-02T09:40:23.674 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:30.309306+0000" 2023-11-02T09:40:23.674 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.674 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.674 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Activating", 2023-11-02T09:40:23.675 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:30.310987+0000", 2023-11-02T09:40:23.675 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:30.753572+0000" 2023-11-02T09:40:23.675 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.675 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.675 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2023-11-02T09:40:23.675 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:30.753572+0000", 2023-11-02T09:40:23.676 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:30.753718+0000" 2023-11-02T09:40:23.676 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.676 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.676 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2023-11-02T09:40:23.676 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:30.753719+0000", 2023-11-02T09:40:23.677 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:30.754101+0000" 2023-11-02T09:40:23.677 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.677 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.677 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Recovering", 2023-11-02T09:40:23.677 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:30.754102+0000", 2023-11-02T09:40:23.678 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:35:00.816681+0000" 2023-11-02T09:40:23.678 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.678 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.678 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/NotRecovering", 2023-11-02T09:40:23.678 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:35:00.816682+0000", 2023-11-02T09:40:23.678 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:35:00.816812+0000" 2023-11-02T09:40:23.679 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.679 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.679 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2023-11-02T09:40:23.680 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:35:00.816815+0000", 2023-11-02T09:40:23.680 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:35:13.082233+0000" 2023-11-02T09:40:23.680 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.680 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.680 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2023-11-02T09:40:23.681 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:35:13.082233+0000", 2023-11-02T09:40:23.681 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:35:13.082521+0000" 2023-11-02T09:40:23.681 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.681 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.681 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Recovering", 2023-11-02T09:40:23.681 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:35:13.082522+0000", 2023-11-02T09:40:23.682 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:36:59.656631+0000" 2023-11-02T09:40:23.682 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.682 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.682 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Recovered", 2023-11-02T09:40:23.682 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:36:59.656632+0000", 2023-11-02T09:40:23.683 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:37:00.222148+0000" 2023-11-02T09:40:23.683 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.683 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.683 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active", 2023-11-02T09:40:23.683 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:30.309307+0000", 2023-11-02T09:40:23.684 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:37:00.222177+0000" 2023-11-02T09:40:23.684 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.684 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.684 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary", 2023-11-02T09:40:23.684 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:29.890781+0000", 2023-11-02T09:40:23.684 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:37:00.222196+0000" 2023-11-02T09:40:23.685 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.685 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.685 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started", 2023-11-02T09:40:23.685 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:29.890764+0000", 2023-11-02T09:40:23.685 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:37:00.222205+0000" 2023-11-02T09:40:23.686 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.686 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.686 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.686 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.686 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "113", 2023-11-02T09:40:23.687 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.687 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.687 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Reset", 2023-11-02T09:40:23.687 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:37:00.222206+0000", 2023-11-02T09:40:23.687 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:37:00.222577+0000" 2023-11-02T09:40:23.687 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.688 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.688 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.688 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.688 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.688 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.689 INFO:tasks.workunit.client.0.smithi116.stdout: "pg": "3.0", 2023-11-02T09:40:23.689 INFO:tasks.workunit.client.0.smithi116.stdout: "currently": "Started/Primary/Active/Clean", 2023-11-02T09:40:23.689 INFO:tasks.workunit.client.0.smithi116.stdout: "history": [ 2023-11-02T09:40:23.689 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.689 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "40", 2023-11-02T09:40:23.689 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.690 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.690 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Start", 2023-11-02T09:40:23.690 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:02.800002+0000", 2023-11-02T09:40:23.690 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:02.800010+0000" 2023-11-02T09:40:23.690 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.691 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.691 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetInfo", 2023-11-02T09:40:23.694 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:02.800027+0000", 2023-11-02T09:40:23.694 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:02.840263+0000" 2023-11-02T09:40:23.694 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.695 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.695 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetLog", 2023-11-02T09:40:23.695 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:02.840263+0000", 2023-11-02T09:40:23.695 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:02.840311+0000" 2023-11-02T09:40:23.695 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.695 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.696 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetMissing", 2023-11-02T09:40:23.696 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:02.840313+0000", 2023-11-02T09:40:23.696 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:02.840323+0000" 2023-11-02T09:40:23.696 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.696 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.697 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2023-11-02T09:40:23.697 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:02.840323+0000", 2023-11-02T09:40:23.697 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:03.076214+0000" 2023-11-02T09:40:23.697 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.697 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.698 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering", 2023-11-02T09:40:23.698 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:02.800013+0000", 2023-11-02T09:40:23.698 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:03.076220+0000" 2023-11-02T09:40:23.698 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.698 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.699 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Activating", 2023-11-02T09:40:23.699 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:03.076411+0000", 2023-11-02T09:40:23.699 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:03.770558+0000" 2023-11-02T09:40:23.699 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.699 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.699 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active", 2023-11-02T09:40:23.700 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:03.076221+0000", 2023-11-02T09:40:23.700 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:03.770573+0000" 2023-11-02T09:40:23.700 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.700 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.700 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary", 2023-11-02T09:40:23.701 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:02.800010+0000", 2023-11-02T09:40:23.701 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:03.770588+0000" 2023-11-02T09:40:23.701 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.701 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.701 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started", 2023-11-02T09:40:23.702 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:02.800000+0000", 2023-11-02T09:40:23.702 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:03.770591+0000" 2023-11-02T09:40:23.702 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.702 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.702 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.702 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.703 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "40", 2023-11-02T09:40:23.703 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.703 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.703 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Reset", 2023-11-02T09:40:23.703 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:03.770592+0000", 2023-11-02T09:40:23.704 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:03.770814+0000" 2023-11-02T09:40:23.704 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.704 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.704 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.704 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.705 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "86", 2023-11-02T09:40:23.705 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.705 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.705 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Start", 2023-11-02T09:40:23.705 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:03.770817+0000", 2023-11-02T09:40:23.705 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:03.770823+0000" 2023-11-02T09:40:23.706 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.706 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.711 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetInfo", 2023-11-02T09:40:23.712 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:03.770828+0000", 2023-11-02T09:40:23.712 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:03.879829+0000" 2023-11-02T09:40:23.712 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.712 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.712 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetLog", 2023-11-02T09:40:23.713 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:03.879829+0000", 2023-11-02T09:40:23.713 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:03.879877+0000" 2023-11-02T09:40:23.713 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.713 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.713 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetMissing", 2023-11-02T09:40:23.714 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:03.879877+0000", 2023-11-02T09:40:23.714 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:03.879888+0000" 2023-11-02T09:40:23.714 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.714 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.714 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2023-11-02T09:40:23.715 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:03.879888+0000", 2023-11-02T09:40:23.715 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:04.181162+0000" 2023-11-02T09:40:23.715 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.715 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.715 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering", 2023-11-02T09:40:23.716 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:03.770826+0000", 2023-11-02T09:40:23.716 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:04.181169+0000" 2023-11-02T09:40:23.716 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.716 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.716 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Activating", 2023-11-02T09:40:23.716 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:04.181298+0000", 2023-11-02T09:40:23.717 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:04.851475+0000" 2023-11-02T09:40:23.717 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.717 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.717 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Recovered", 2023-11-02T09:40:23.717 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:04.851476+0000", 2023-11-02T09:40:23.718 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:04.851511+0000" 2023-11-02T09:40:23.718 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.718 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.718 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Clean", 2023-11-02T09:40:23.718 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:04.851511+0000", 2023-11-02T09:40:23.719 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:42.256674+0000" 2023-11-02T09:40:23.719 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.719 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.719 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active", 2023-11-02T09:40:23.719 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:04.181170+0000", 2023-11-02T09:40:23.719 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:42.256684+0000" 2023-11-02T09:40:23.720 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.720 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.720 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary", 2023-11-02T09:40:23.721 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:03.770824+0000", 2023-11-02T09:40:23.721 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:42.256697+0000" 2023-11-02T09:40:23.721 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.721 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.721 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started", 2023-11-02T09:40:23.721 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:03.770814+0000", 2023-11-02T09:40:23.722 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:42.256699+0000" 2023-11-02T09:40:23.722 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.722 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.722 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.722 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.723 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "86", 2023-11-02T09:40:23.723 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.723 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.723 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Reset", 2023-11-02T09:40:23.723 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:42.256700+0000", 2023-11-02T09:40:23.724 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:42.256915+0000" 2023-11-02T09:40:23.724 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.724 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.724 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.724 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.724 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "105", 2023-11-02T09:40:23.725 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.725 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.725 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Start", 2023-11-02T09:40:23.725 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:42.256918+0000", 2023-11-02T09:40:23.725 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:42.256927+0000" 2023-11-02T09:40:23.726 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.726 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.726 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetInfo", 2023-11-02T09:40:23.726 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:42.256933+0000", 2023-11-02T09:40:23.727 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:42.257020+0000" 2023-11-02T09:40:23.727 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.727 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.727 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetLog", 2023-11-02T09:40:23.727 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:42.257021+0000", 2023-11-02T09:40:23.728 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:42.257070+0000" 2023-11-02T09:40:23.728 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.728 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.728 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetMissing", 2023-11-02T09:40:23.728 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:42.257070+0000", 2023-11-02T09:40:23.729 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:42.257081+0000" 2023-11-02T09:40:23.729 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.729 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.729 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2023-11-02T09:40:23.729 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:42.257082+0000", 2023-11-02T09:40:23.730 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:42.600672+0000" 2023-11-02T09:40:23.730 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.730 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.730 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering", 2023-11-02T09:40:23.730 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:42.256930+0000", 2023-11-02T09:40:23.730 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:42.600679+0000" 2023-11-02T09:40:23.731 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.731 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.731 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Activating", 2023-11-02T09:40:23.731 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:42.600740+0000", 2023-11-02T09:40:23.731 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:42.650993+0000" 2023-11-02T09:40:23.732 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.732 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.732 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Recovered", 2023-11-02T09:40:23.732 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:42.650994+0000", 2023-11-02T09:40:23.732 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:29:42.651020+0000" 2023-11-02T09:40:23.732 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.733 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.733 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Clean", 2023-11-02T09:40:23.733 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:42.651020+0000", 2023-11-02T09:40:23.733 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:47.384030+0000" 2023-11-02T09:40:23.733 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.734 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.734 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active", 2023-11-02T09:40:23.734 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:42.600681+0000", 2023-11-02T09:40:23.734 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:47.384057+0000" 2023-11-02T09:40:23.734 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.735 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.735 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary", 2023-11-02T09:40:23.735 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:42.256928+0000", 2023-11-02T09:40:23.735 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:47.384234+0000" 2023-11-02T09:40:23.735 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.735 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.736 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started", 2023-11-02T09:40:23.736 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:29:42.256916+0000", 2023-11-02T09:40:23.736 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:47.384237+0000" 2023-11-02T09:40:23.736 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.736 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.737 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.737 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.737 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "106", 2023-11-02T09:40:23.737 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.737 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.738 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Reset", 2023-11-02T09:40:23.738 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:47.384238+0000", 2023-11-02T09:40:23.738 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:47.384648+0000" 2023-11-02T09:40:23.738 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.738 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.738 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.739 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.739 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "107", 2023-11-02T09:40:23.739 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.739 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.739 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Start", 2023-11-02T09:40:23.740 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:47.384654+0000", 2023-11-02T09:40:23.740 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:47.384665+0000" 2023-11-02T09:40:23.740 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.740 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.740 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetInfo", 2023-11-02T09:40:23.741 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:47.384672+0000", 2023-11-02T09:40:23.741 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:47.406394+0000" 2023-11-02T09:40:23.741 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.741 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.741 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetLog", 2023-11-02T09:40:23.741 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:47.406395+0000", 2023-11-02T09:40:23.742 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:47.406480+0000" 2023-11-02T09:40:23.742 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.742 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.742 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering", 2023-11-02T09:40:23.742 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:47.384669+0000", 2023-11-02T09:40:23.743 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:47.406493+0000" 2023-11-02T09:40:23.743 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.743 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.743 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/WaitActingChange", 2023-11-02T09:40:23.743 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:47.406493+0000", 2023-11-02T09:40:23.744 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:47.640755+0000" 2023-11-02T09:40:23.744 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.744 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.744 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary", 2023-11-02T09:40:23.744 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:47.384666+0000", 2023-11-02T09:40:23.744 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:47.640794+0000" 2023-11-02T09:40:23.745 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.745 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.745 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started", 2023-11-02T09:40:23.745 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:47.384649+0000", 2023-11-02T09:40:23.745 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:47.640799+0000" 2023-11-02T09:40:23.746 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.746 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.746 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.746 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.746 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "107", 2023-11-02T09:40:23.747 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.747 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.747 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Reset", 2023-11-02T09:40:23.747 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:47.640800+0000", 2023-11-02T09:40:23.747 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:47.641097+0000" 2023-11-02T09:40:23.748 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.748 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.748 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.748 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.749 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "117", 2023-11-02T09:40:23.749 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.749 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.749 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Start", 2023-11-02T09:40:23.749 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:47.641102+0000", 2023-11-02T09:40:23.749 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:47.641115+0000" 2023-11-02T09:40:23.750 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.750 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.750 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetInfo", 2023-11-02T09:40:23.750 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:47.641124+0000", 2023-11-02T09:40:23.750 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:47.749857+0000" 2023-11-02T09:40:23.751 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.751 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.751 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetLog", 2023-11-02T09:40:23.751 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:47.749857+0000", 2023-11-02T09:40:23.751 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:47.749937+0000" 2023-11-02T09:40:23.752 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.752 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.752 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetMissing", 2023-11-02T09:40:23.752 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:47.749937+0000", 2023-11-02T09:40:23.753 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:47.749953+0000" 2023-11-02T09:40:23.753 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.753 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.753 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2023-11-02T09:40:23.754 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:47.749953+0000", 2023-11-02T09:40:23.754 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:48.109943+0000" 2023-11-02T09:40:23.754 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.754 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.754 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering", 2023-11-02T09:40:23.755 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:47.641119+0000", 2023-11-02T09:40:23.755 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:48.109953+0000" 2023-11-02T09:40:23.755 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.755 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.755 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Activating", 2023-11-02T09:40:23.755 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:48.111471+0000", 2023-11-02T09:40:23.756 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:48.269866+0000" 2023-11-02T09:40:23.756 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.756 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.756 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2023-11-02T09:40:23.756 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:48.269867+0000", 2023-11-02T09:40:23.757 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:35:00.816778+0000" 2023-11-02T09:40:23.757 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.757 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.757 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2023-11-02T09:40:23.757 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:35:00.816778+0000", 2023-11-02T09:40:23.758 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:35:00.817097+0000" 2023-11-02T09:40:23.758 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.758 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.758 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Recovering", 2023-11-02T09:40:23.758 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:35:00.817100+0000", 2023-11-02T09:40:23.759 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:35:13.082131+0000" 2023-11-02T09:40:23.759 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.759 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.759 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/NotRecovering", 2023-11-02T09:40:23.759 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:35:13.082132+0000", 2023-11-02T09:40:23.759 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:35:13.082292+0000" 2023-11-02T09:40:23.760 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.760 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.760 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2023-11-02T09:40:23.760 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:35:13.082293+0000", 2023-11-02T09:40:23.760 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:38:41.060948+0000" 2023-11-02T09:40:23.761 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.761 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.761 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2023-11-02T09:40:23.761 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:38:41.060949+0000", 2023-11-02T09:40:23.761 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:38:41.061206+0000" 2023-11-02T09:40:23.762 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.762 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.762 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Recovering", 2023-11-02T09:40:23.762 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:38:41.061206+0000", 2023-11-02T09:40:23.762 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:40:20.382799+0000" 2023-11-02T09:40:23.762 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.763 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.763 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Recovered", 2023-11-02T09:40:23.763 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:40:20.382800+0000", 2023-11-02T09:40:23.763 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:40:20.676040+0000" 2023-11-02T09:40:23.763 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.764 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.764 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active", 2023-11-02T09:40:23.764 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:48.109954+0000", 2023-11-02T09:40:23.764 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:40:20.676059+0000" 2023-11-02T09:40:23.764 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.765 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.765 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary", 2023-11-02T09:40:23.765 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:47.641115+0000", 2023-11-02T09:40:23.765 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:40:20.676079+0000" 2023-11-02T09:40:23.765 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.765 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.766 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started", 2023-11-02T09:40:23.766 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:47.641097+0000", 2023-11-02T09:40:23.766 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:40:20.676085+0000" 2023-11-02T09:40:23.766 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.766 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.767 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.767 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.767 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "117", 2023-11-02T09:40:23.767 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.767 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.768 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Reset", 2023-11-02T09:40:23.768 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:40:20.676085+0000", 2023-11-02T09:40:23.768 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:40:20.676451+0000" 2023-11-02T09:40:23.768 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.768 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.769 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.769 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.769 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.769 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.769 INFO:tasks.workunit.client.0.smithi116.stdout: "pg": "1.0", 2023-11-02T09:40:23.769 INFO:tasks.workunit.client.0.smithi116.stdout: "currently": "Started/Primary/Active/Clean", 2023-11-02T09:40:23.770 INFO:tasks.workunit.client.0.smithi116.stdout: "history": [ 2023-11-02T09:40:23.770 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.770 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "27", 2023-11-02T09:40:23.770 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.770 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.771 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Initial", 2023-11-02T09:40:23.771 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:57.086518+0000", 2023-11-02T09:40:23.771 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:57.086665+0000" 2023-11-02T09:40:23.771 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.771 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.772 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.772 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.772 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "27", 2023-11-02T09:40:23.772 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.772 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.772 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Reset", 2023-11-02T09:40:23.773 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:57.086666+0000", 2023-11-02T09:40:23.773 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:57.086692+0000" 2023-11-02T09:40:23.773 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.773 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.773 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.774 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.774 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "29", 2023-11-02T09:40:23.774 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.774 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.774 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Start", 2023-11-02T09:40:23.775 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:57.086695+0000", 2023-11-02T09:40:23.775 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:57.086702+0000" 2023-11-02T09:40:23.775 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.775 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.775 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetInfo", 2023-11-02T09:40:23.775 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:57.086707+0000", 2023-11-02T09:40:23.776 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:57.170180+0000" 2023-11-02T09:40:23.776 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.776 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.776 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetLog", 2023-11-02T09:40:23.776 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:57.170181+0000", 2023-11-02T09:40:23.777 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:57.170233+0000" 2023-11-02T09:40:23.777 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.777 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.777 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetMissing", 2023-11-02T09:40:23.777 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:57.170233+0000", 2023-11-02T09:40:23.777 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:57.170242+0000" 2023-11-02T09:40:23.778 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.778 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.778 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2023-11-02T09:40:23.778 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:57.170242+0000", 2023-11-02T09:40:23.778 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:57.329094+0000" 2023-11-02T09:40:23.779 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.779 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.779 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering", 2023-11-02T09:40:23.779 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:57.086704+0000", 2023-11-02T09:40:23.779 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:57.329100+0000" 2023-11-02T09:40:23.780 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.780 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.780 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Activating", 2023-11-02T09:40:23.780 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:57.329269+0000", 2023-11-02T09:40:23.780 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:58.015106+0000" 2023-11-02T09:40:23.780 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.781 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.781 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Recovered", 2023-11-02T09:40:23.781 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:58.015107+0000", 2023-11-02T09:40:23.781 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:58.015163+0000" 2023-11-02T09:40:23.781 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.782 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.782 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Clean", 2023-11-02T09:40:23.782 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:58.015163+0000", 2023-11-02T09:40:23.782 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:58.408649+0000" 2023-11-02T09:40:23.782 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.783 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.783 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active", 2023-11-02T09:40:23.783 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:57.329101+0000", 2023-11-02T09:40:23.783 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:58.408665+0000" 2023-11-02T09:40:23.783 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.783 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.784 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary", 2023-11-02T09:40:23.784 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:57.086702+0000", 2023-11-02T09:40:23.784 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:58.408678+0000" 2023-11-02T09:40:23.784 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.784 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.785 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started", 2023-11-02T09:40:23.785 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:57.086693+0000", 2023-11-02T09:40:23.785 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:58.408680+0000" 2023-11-02T09:40:23.785 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.785 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.786 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.786 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.786 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "29", 2023-11-02T09:40:23.786 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.786 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.787 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Reset", 2023-11-02T09:40:23.787 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:58.408681+0000", 2023-11-02T09:40:23.787 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:58.408918+0000" 2023-11-02T09:40:23.787 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.787 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.788 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.788 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.788 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "101", 2023-11-02T09:40:23.788 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.788 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.789 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Start", 2023-11-02T09:40:23.789 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:58.408921+0000", 2023-11-02T09:40:23.789 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:58.408931+0000" 2023-11-02T09:40:23.789 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.789 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.789 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetInfo", 2023-11-02T09:40:23.790 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:58.408937+0000", 2023-11-02T09:40:23.790 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:58.518337+0000" 2023-11-02T09:40:23.790 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.791 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.791 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetLog", 2023-11-02T09:40:23.791 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:58.518338+0000", 2023-11-02T09:40:23.791 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:58.518402+0000" 2023-11-02T09:40:23.791 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.791 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.792 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetMissing", 2023-11-02T09:40:23.792 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:58.518402+0000", 2023-11-02T09:40:23.792 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:58.518413+0000" 2023-11-02T09:40:23.792 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.792 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.793 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2023-11-02T09:40:23.793 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:58.518413+0000", 2023-11-02T09:40:23.793 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:59.112542+0000" 2023-11-02T09:40:23.793 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.793 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.794 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering", 2023-11-02T09:40:23.794 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:58.408934+0000", 2023-11-02T09:40:23.794 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:59.112558+0000" 2023-11-02T09:40:23.794 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.795 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.795 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Activating", 2023-11-02T09:40:23.795 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:59.112674+0000", 2023-11-02T09:40:23.795 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:59.632739+0000" 2023-11-02T09:40:23.795 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.795 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.796 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Recovered", 2023-11-02T09:40:23.796 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:59.632739+0000", 2023-11-02T09:40:23.796 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:28:59.632771+0000" 2023-11-02T09:40:23.796 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.796 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.797 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Clean", 2023-11-02T09:40:23.797 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:59.632771+0000", 2023-11-02T09:40:23.797 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:38.991258+0000" 2023-11-02T09:40:23.797 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.797 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.797 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active", 2023-11-02T09:40:23.798 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:59.112559+0000", 2023-11-02T09:40:23.798 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:38.991302+0000" 2023-11-02T09:40:23.798 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.798 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.798 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary", 2023-11-02T09:40:23.799 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:58.408931+0000", 2023-11-02T09:40:23.799 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:38.991437+0000" 2023-11-02T09:40:23.799 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.799 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.799 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started", 2023-11-02T09:40:23.800 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:28:58.408918+0000", 2023-11-02T09:40:23.800 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:38.991442+0000" 2023-11-02T09:40:23.800 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.800 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.800 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.800 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.801 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "101", 2023-11-02T09:40:23.801 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.801 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.801 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Reset", 2023-11-02T09:40:23.801 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:38.991443+0000", 2023-11-02T09:40:23.802 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:38.992058+0000" 2023-11-02T09:40:23.802 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.802 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.802 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.802 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.803 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "103", 2023-11-02T09:40:23.803 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.803 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.803 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Start", 2023-11-02T09:40:23.803 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:38.992075+0000", 2023-11-02T09:40:23.803 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:38.992099+0000" 2023-11-02T09:40:23.804 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.804 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.804 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetInfo", 2023-11-02T09:40:23.804 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:38.992115+0000", 2023-11-02T09:40:23.804 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:39.109110+0000" 2023-11-02T09:40:23.805 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.805 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.805 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetLog", 2023-11-02T09:40:23.805 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:39.109111+0000", 2023-11-02T09:40:23.805 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:39.109178+0000" 2023-11-02T09:40:23.805 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.806 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.806 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering", 2023-11-02T09:40:23.806 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:38.992108+0000", 2023-11-02T09:40:23.806 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:39.109188+0000" 2023-11-02T09:40:23.806 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.807 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.807 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/WaitActingChange", 2023-11-02T09:40:23.807 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:39.109189+0000", 2023-11-02T09:40:23.807 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:39.686873+0000" 2023-11-02T09:40:23.807 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.808 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.808 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary", 2023-11-02T09:40:23.808 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:38.992100+0000", 2023-11-02T09:40:23.808 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:39.686906+0000" 2023-11-02T09:40:23.808 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.808 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.809 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started", 2023-11-02T09:40:23.809 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:38.992059+0000", 2023-11-02T09:40:23.809 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:39.686911+0000" 2023-11-02T09:40:23.809 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.809 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.810 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.810 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.810 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "103", 2023-11-02T09:40:23.810 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.810 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.811 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Reset", 2023-11-02T09:40:23.811 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:39.686913+0000", 2023-11-02T09:40:23.811 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:39.687252+0000" 2023-11-02T09:40:23.811 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.811 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.812 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.812 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.812 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "115", 2023-11-02T09:40:23.812 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.812 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.813 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Start", 2023-11-02T09:40:23.813 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:39.687257+0000", 2023-11-02T09:40:23.813 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:39.687272+0000" 2023-11-02T09:40:23.813 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.813 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.814 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetInfo", 2023-11-02T09:40:23.814 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:39.687281+0000", 2023-11-02T09:40:23.814 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:39.804769+0000" 2023-11-02T09:40:23.814 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.814 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.815 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetLog", 2023-11-02T09:40:23.815 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:39.804769+0000", 2023-11-02T09:40:23.815 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:39.804852+0000" 2023-11-02T09:40:23.815 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.815 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.815 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/GetMissing", 2023-11-02T09:40:23.816 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:39.804852+0000", 2023-11-02T09:40:23.816 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:39.804868+0000" 2023-11-02T09:40:23.816 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.816 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.816 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2023-11-02T09:40:23.817 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:39.804868+0000", 2023-11-02T09:40:23.817 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:40.139447+0000" 2023-11-02T09:40:23.817 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.817 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.817 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Peering", 2023-11-02T09:40:23.818 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:39.687276+0000", 2023-11-02T09:40:23.818 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:40.139459+0000" 2023-11-02T09:40:23.818 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.818 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.818 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Activating", 2023-11-02T09:40:23.818 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:40.141196+0000", 2023-11-02T09:40:23.819 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:34:40.408608+0000" 2023-11-02T09:40:23.819 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.819 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.819 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2023-11-02T09:40:23.819 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:40.408609+0000", 2023-11-02T09:40:23.820 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:36:59.656827+0000" 2023-11-02T09:40:23.820 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.820 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.820 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2023-11-02T09:40:23.820 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:36:59.656827+0000", 2023-11-02T09:40:23.821 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:36:59.657148+0000" 2023-11-02T09:40:23.821 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.821 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.821 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Recovering", 2023-11-02T09:40:23.821 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:36:59.657151+0000", 2023-11-02T09:40:23.822 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:38:41.060793+0000" 2023-11-02T09:40:23.822 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.822 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.822 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active/Recovered", 2023-11-02T09:40:23.822 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:38:41.060794+0000", 2023-11-02T09:40:23.823 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:38:41.411887+0000" 2023-11-02T09:40:23.823 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.823 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.823 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary/Active", 2023-11-02T09:40:23.823 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:40.139459+0000", 2023-11-02T09:40:23.824 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:38:41.411914+0000" 2023-11-02T09:40:23.824 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.824 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.824 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started/Primary", 2023-11-02T09:40:23.824 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:39.687272+0000", 2023-11-02T09:40:23.825 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:38:41.411935+0000" 2023-11-02T09:40:23.825 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.825 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.825 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Started", 2023-11-02T09:40:23.825 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:34:39.687253+0000", 2023-11-02T09:40:23.825 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:38:41.411938+0000" 2023-11-02T09:40:23.826 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.826 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.826 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2023-11-02T09:40:23.826 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.826 INFO:tasks.workunit.client.0.smithi116.stdout: "epoch": "115", 2023-11-02T09:40:23.827 INFO:tasks.workunit.client.0.smithi116.stdout: "states": [ 2023-11-02T09:40:23.827 INFO:tasks.workunit.client.0.smithi116.stdout: { 2023-11-02T09:40:23.827 INFO:tasks.workunit.client.0.smithi116.stdout: "state": "Reset", 2023-11-02T09:40:23.827 INFO:tasks.workunit.client.0.smithi116.stdout: "enter": "2023-11-02T09:38:41.411939+0000", 2023-11-02T09:40:23.827 INFO:tasks.workunit.client.0.smithi116.stdout: "exit": "2023-11-02T09:38:41.412302+0000" 2023-11-02T09:40:23.827 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.828 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.828 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.828 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.828 INFO:tasks.workunit.client.0.smithi116.stdout: } 2023-11-02T09:40:23.828 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2023-11-02T09:40:23.829 INFO:tasks.workunit.client.0.smithi116.stdout:} 2023-11-02T09:40:23.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:302: TEST_recovery_priority: '[' 0 '!=' 0 ']' 2023-11-02T09:40:23.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:306: TEST_recovery_priority: echo TEST PASSED 2023-11-02T09:40:23.830 INFO:tasks.workunit.client.0.smithi116.stdout:TEST PASSED 2023-11-02T09:40:23.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:309: TEST_recovery_priority: delete_pool test1 2023-11-02T09:40:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2023-11-02T09:40:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2023-11-02T09:40:24.042 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test1' does not exist 2023-11-02T09:40:24.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:310: TEST_recovery_priority: delete_pool test3 2023-11-02T09:40:24.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2023-11-02T09:40:24.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2023-11-02T09:40:24.729 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test3' does not exist 2023-11-02T09:40:24.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:311: TEST_recovery_priority: delete_pool test7 2023-11-02T09:40:24.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test7 2023-11-02T09:40:24.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2023-11-02T09:40:25.395 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test7' does not exist 2023-11-02T09:40:25.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:312: TEST_recovery_priority: kill_daemons td/osd-recovery-prio 2023-11-02T09:40:25.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:40:25.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:40:25.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:40:25.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:40:25.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:40:30.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:40:30.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:313: TEST_recovery_priority: return 0 2023-11-02T09:40:30.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:41: run: teardown td/osd-recovery-prio 2023-11-02T09:40:30.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2023-11-02T09:40:30.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:40:30.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2023-11-02T09:40:30.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:40:30.723 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:40:30.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:40:30.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:40:30.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:40:30.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:40:30.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:40:30.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:40:30.731 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:40:30.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:40:30.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:40:30.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:40:30.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:40:30.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:40:30.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:40:30.734 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:40:30.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:40:30.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:40:30.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2023-11-02T09:40:30.806 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:40:30.807 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:40:30.807 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:40:30.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.261297 2023-11-02T09:40:30.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:40:30.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:40:30.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2023-11-02T09:40:30.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-recovery-prio 0 2023-11-02T09:40:30.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2023-11-02T09:40:30.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-11-02T09:40:30.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2023-11-02T09:40:30.810 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:40:30.810 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:40:30.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:40:30.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:40:30.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:40:30.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:40:30.812 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:40:30.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:40:30.812 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:40:30.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:40:30.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:40:30.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:40:30.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:40:30.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:40:30.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:40:30.814 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:40:30.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:40:30.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-11-02T09:40:30.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2023-11-02T09:40:30.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:40:30.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:40:30.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.261297 2023-11-02T09:40:30.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.261297 2023-11-02T09:40:30.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:40:30.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:40:30.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2023-11-02T09:40:30.825 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-02T09:40:30.825 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-02T09:40:30.857 INFO:tasks.workunit:Running workunit osd/osd-recovery-space.sh... 2023-11-02T09:40:30.857 DEBUG:teuthology.orchestra.run.smithi116:workunit test osd/osd-recovery-space.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh 2023-11-02T09:40:30.919 INFO:tasks.workunit.client.0.smithi116.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-11-02T09:40:30.923 INFO:tasks.workunit.client.0.smithi116.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-11-02T09:40:30.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T09:40:30.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-11-02T09:40:30.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2023-11-02T09:40:30.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2023-11-02T09:40:30.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2023-11-02T09:40:30.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2023-11-02T09:40:30.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2023-11-02T09:40:30.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2023-11-02T09:40:30.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-recovery-space 2023-11-02T09:40:30.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:21: run: local dir=td/osd-recovery-space 2023-11-02T09:40:30.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:22: run: shift 2023-11-02T09:40:30.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: export CEPH_MON=127.0.0.1:7221 2023-11-02T09:40:30.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: CEPH_MON=127.0.0.1:7221 2023-11-02T09:40:30.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:25: run: export CEPH_ARGS 2023-11-02T09:40:30.926 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: uuidgen 2023-11-02T09:40:30.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: CEPH_ARGS+='--fsid=e9475e6c-4956-4813-8dbd-b1d4bc7eb092 --auth-supported=none ' 2023-11-02T09:40:30.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7221 ' 2023-11-02T09:40:30.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:28: run: CEPH_ARGS+='--osd_max_backfills=10 ' 2023-11-02T09:40:30.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2023-11-02T09:40:30.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: export objects=600 2023-11-02T09:40:30.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: objects=600 2023-11-02T09:40:30.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:31: run: export poolprefix=test 2023-11-02T09:40:30.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:31: run: poolprefix=test 2023-11-02T09:40:30.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: set 2023-11-02T09:40:30.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-11-02T09:40:30.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: local funcs=TEST_recovery_test_simple 2023-11-02T09:40:30.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:34: run: for func in $funcs 2023-11-02T09:40:30.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:35: run: setup td/osd-recovery-space 2023-11-02T09:40:30.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-space 2023-11-02T09:40:30.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-space 2023-11-02T09:40:30.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2023-11-02T09:40:30.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-11-02T09:40:30.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2023-11-02T09:40:30.933 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:40:30.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:40:30.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:40:30.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:40:30.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:40:30.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:40:30.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:40:30.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:40:30.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:40:30.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:40:30.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:40:30.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:40:30.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:40:30.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:40:30.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:40:30.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:40:30.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:40:30.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-11-02T09:40:30.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2023-11-02T09:40:30.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:40:30.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:40:30.947 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.316946 2023-11-02T09:40:30.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.316946 2023-11-02T09:40:30.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:40:30.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:40:30.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-space 2023-11-02T09:40:30.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-11-02T09:40:30.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:40:30.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.316946 2023-11-02T09:40:30.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.316946 2023-11-02T09:40:30.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-11-02T09:40:30.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-11-02T09:40:30.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-11-02T09:40:30.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-11-02T09:40:30.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-space 1' TERM HUP INT 2023-11-02T09:40:30.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:36: run: TEST_recovery_test_simple td/osd-recovery-space 2023-11-02T09:40:30.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:89: TEST_recovery_test_simple: local dir=td/osd-recovery-space 2023-11-02T09:40:30.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:90: TEST_recovery_test_simple: local pools=1 2023-11-02T09:40:30.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:91: TEST_recovery_test_simple: local OSDS=2 2023-11-02T09:40:30.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:93: TEST_recovery_test_simple: run_mon td/osd-recovery-space a 2023-11-02T09:40:30.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-space 2023-11-02T09:40:30.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-11-02T09:40:30.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-11-02T09:40:30.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-11-02T09:40:30.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-space/a 2023-11-02T09:40:30.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-space/a --run-dir=td/osd-recovery-space 2023-11-02T09:40:31.334 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-11-02T09:40:31.334 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:40:31.334 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:40:31.335 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:40:31.335 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:40:31.335 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.316946 2023-11-02T09:40:31.335 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.316946/$cluster-$name.asok' 2023-11-02T09:40:31.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-space/a '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.316946/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-space/log --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-11-02T09:40:31.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-11-02T09:40:31.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-11-02T09:40:31.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:40:31.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:40:31.521 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-11-02T09:40:31.521 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-11-02T09:40:31.521 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:40:31.521 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:40:31.521 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:40:31.522 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:40:31.522 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:40:31.522 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.316946 2023-11-02T09:40:31.522 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.316946/ceph-mon.a.asok 2023-11-02T09:40:31.524 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:40:31.525 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.316946/ceph-mon.a.asok config get fsid 2023-11-02T09:40:31.600 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-11-02T09:40:31.600 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-11-02T09:40:31.600 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-11-02T09:40:31.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-11-02T09:40:31.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-11-02T09:40:31.601 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-11-02T09:40:31.601 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-11-02T09:40:31.601 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-11-02T09:40:31.601 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-11-02T09:40:31.602 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:40:31.602 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.316946 2023-11-02T09:40:31.602 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.316946/ceph-mon.a.asok 2023-11-02T09:40:31.603 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-11-02T09:40:31.604 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.316946/ceph-mon.a.asok config get mon_host 2023-11-02T09:40:31.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:94: TEST_recovery_test_simple: run_mgr td/osd-recovery-space x 2023-11-02T09:40:31.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-space 2023-11-02T09:40:31.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-11-02T09:40:31.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-11-02T09:40:31.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-11-02T09:40:31.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-space/x 2023-11-02T09:40:31.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-11-02T09:40:32.523 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-11-02T09:40:32.523 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:40:32.523 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:40:32.524 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:40:32.524 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:40:32.524 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.316946 2023-11-02T09:40:32.524 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.316946/$cluster-$name.asok' 2023-11-02T09:40:32.524 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:40:32.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-space/x '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.316946/$cluster-$name.asok' --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-11-02T09:40:32.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:95: TEST_recovery_test_simple: export CEPH_ARGS 2023-11-02T09:40:32.551 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: expr 2 - 1 2023-11-02T09:40:32.552 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: seq 0 1 2023-11-02T09:40:32.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-11-02T09:40:32.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:99: TEST_recovery_test_simple: run_osd td/osd-recovery-space 0 2023-11-02T09:40:32.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-space 2023-11-02T09:40:32.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:40:32.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-11-02T09:40:32.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:40:32.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-space/0 2023-11-02T09:40:32.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e9475e6c-4956-4813-8dbd-b1d4bc7eb092 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true ' 2023-11-02T09:40:32.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:40:32.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:40:32.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:40:32.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/0' 2023-11-02T09:40:32.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/0/journal' 2023-11-02T09:40:32.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:40:32.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:40:32.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2023-11-02T09:40:32.561 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:40:32.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:40:32.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:40:32.563 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:40:32.563 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:40:32.563 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.316946 2023-11-02T09:40:32.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.316946/$cluster-$name.asok' 2023-11-02T09:40:32.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.316946/$cluster-$name.asok' 2023-11-02T09:40:32.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:40:32.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:40:32.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:40:32.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2023-11-02T09:40:32.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2023-11-02T09:40:32.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:40:32.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:40:32.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:40:32.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:40:32.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:40:32.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:40:32.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-space/0 2023-11-02T09:40:32.568 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:40:32.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5673e3cc-e9e3-4b72-aef0-c3e717faa391 2023-11-02T09:40:32.568 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 5673e3cc-e9e3-4b72-aef0-c3e717faa391 2023-11-02T09:40:32.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5673e3cc-e9e3-4b72-aef0-c3e717faa391' 2023-11-02T09:40:32.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:40:32.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCQbkNl9tkUIhAAGZE1oLDx/gWQN33B/JvEuQ== 2023-11-02T09:40:32.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCQbkNl9tkUIhAAGZE1oLDx/gWQN33B/JvEuQ=="}' 2023-11-02T09:40:32.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5673e3cc-e9e3-4b72-aef0-c3e717faa391 -i td/osd-recovery-space/0/new.json 2023-11-02T09:40:32.919 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:40:32.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-space/0/new.json 2023-11-02T09:40:32.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=e9475e6c-4956-4813-8dbd-b1d4bc7eb092 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.316946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCQbkNl9tkUIhAAGZE1oLDx/gWQN33B/JvEuQ== --osd-uuid 5673e3cc-e9e3-4b72-aef0-c3e717faa391 2023-11-02T09:40:32.959 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:40:32.956+0000 7f4a81a0e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:40:32.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:40:32.958+0000 7f4a81a0e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:40:32.962 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:40:32.959+0000 7f4a81a0e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:40:32.963 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:40:32.960+0000 7f4a81a0e500 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2023-11-02T09:40:32.963 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:40:32.960+0000 7f4a81a0e500 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2023-11-02T09:40:32.963 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:40:32.960+0000 7f4a81a0e500 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2023-11-02T09:40:33.034 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:40:33.032+0000 7f4a81a0e500 -1 bluestore(td/osd-recovery-space/0) _read_fsid unparsable uuid 2023-11-02T09:40:38.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-space/0/keyring 2023-11-02T09:40:38.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:40:38.726 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2023-11-02T09:40:38.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-11-02T09:40:38.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:40:39.245 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2023-11-02T09:40:39.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-11-02T09:40:39.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=e9475e6c-4956-4813-8dbd-b1d4bc7eb092 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.316946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:40:39.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:40:39.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:40:39.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:40:39.273 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:40:39.270+0000 7f4115c9c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:40:39.290 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:40:39.287+0000 7f4115c9c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:40:39.291 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:40:39.288+0000 7f4115c9c500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:40:39.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-11-02T09:40:39.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:40:39.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-11-02T09:40:39.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:40:39.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:40:39.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:40:39.696 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:40:39.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:40:39.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:40:39.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:40:39.836 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:40:39.834+0000 7f4115c9c500 -1 Falling back to public interface 2023-11-02T09:40:40.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:40:41.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:40:41.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:40:41.071 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:40:41.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:40:41.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:40:41.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:40:41.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:40:42.083 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:40:42.080+0000 7f4115c9c500 -1 osd.0 0 log_to_monitors true 2023-11-02T09:40:42.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:40:42.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:40:42.491 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:40:42.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:40:42.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:40:42.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:40:42.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:40:43.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:40:43.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:40:43.871 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:40:43.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:40:43.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:40:43.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:40:44.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:40:45.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:40:45.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:40:45.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:40:45.270 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:40:45.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:40:45.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:40:45.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:40:46.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:40:46.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:40:46.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:40:46.705 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:40:46.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:40:46.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:40:47.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:40:48.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:40:48.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:40:48.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:40:48.109 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:40:48.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:40:48.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:40:48.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:40:49.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:40:49.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:40:49.523 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:40:49.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:40:49.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:40:49.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:40:50.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:40:51.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:40:51.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:40:51.019 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:40:51.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:40:51.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:40:51.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:40:51.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:40:52.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:40:52.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:40:52.425 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:40:52.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:40:52.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:40:52.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:40:52.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:40:53.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:40:53.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:40:53.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:40:53.884 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:40:53.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:40:53.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:40:54.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:40:55.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:40:55.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:40:55.263 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:40:55.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:40:55.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:40:55.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:40:56.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:40:57.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:40:57.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:40:57.013 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:40:57.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:40:57.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:40:57.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:40:57.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:40:58.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:40:58.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:40:58.452 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:40:58.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:40:58.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:40:58.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:40:58.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:40:59.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:40:59.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:40:59.912 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:40:59.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:40:59.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:40:59.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:41:00.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:01.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:01.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:01.578 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:41:01.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:41:01.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:01.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:41:02.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:03.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:03.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:03.363 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:41:03.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:41:03.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:03.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:41:03.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:04.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:04.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:04.976 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T09:41:04.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T09:41:04.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:04.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:41:05.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:06.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:06.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:06.494 INFO:tasks.workunit.client.0.smithi116.stdout:18 2023-11-02T09:41:06.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2023-11-02T09:41:06.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:06.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-11-02T09:41:06.897 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/318032,v1:127.0.0.1:6803/318032] [v2:127.0.0.1:6804/318032,v1:127.0.0.1:6805/318032] exists,up 5673e3cc-e9e3-4b72-aef0-c3e717faa391 2023-11-02T09:41:06.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:41:06.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:41:06.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:41:06.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-11-02T09:41:06.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:99: TEST_recovery_test_simple: run_osd td/osd-recovery-space 1 2023-11-02T09:41:06.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-space 2023-11-02T09:41:06.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-11-02T09:41:06.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-11-02T09:41:06.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-11-02T09:41:06.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-space/1 2023-11-02T09:41:06.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e9475e6c-4956-4813-8dbd-b1d4bc7eb092 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true ' 2023-11-02T09:41:06.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-11-02T09:41:06.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-11-02T09:41:06.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-11-02T09:41:06.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/1' 2023-11-02T09:41:06.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/1/journal' 2023-11-02T09:41:06.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-11-02T09:41:06.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-11-02T09:41:06.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2023-11-02T09:41:06.912 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-11-02T09:41:06.913 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-11-02T09:41:06.913 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-11-02T09:41:06.913 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-11-02T09:41:06.913 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:41:06.913 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.316946 2023-11-02T09:41:06.914 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.316946/$cluster-$name.asok' 2023-11-02T09:41:06.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.316946/$cluster-$name.asok' 2023-11-02T09:41:06.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-11-02T09:41:06.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-11-02T09:41:06.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-11-02T09:41:06.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2023-11-02T09:41:06.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2023-11-02T09:41:06.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-11-02T09:41:06.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-11-02T09:41:06.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-11-02T09:41:06.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-11-02T09:41:06.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-11-02T09:41:06.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-11-02T09:41:06.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-space/1 2023-11-02T09:41:06.916 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-11-02T09:41:06.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5674d109-af32-4798-a8ce-0690521b463f 2023-11-02T09:41:06.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5674d109-af32-4798-a8ce-0690521b463f' 2023-11-02T09:41:06.917 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 5674d109-af32-4798-a8ce-0690521b463f 2023-11-02T09:41:06.918 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-11-02T09:41:06.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCybkNlXgqcNhAA/A2OGL8XeHekRpVCd5ro+A== 2023-11-02T09:41:06.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCybkNlXgqcNhAA/A2OGL8XeHekRpVCd5ro+A=="}' 2023-11-02T09:41:06.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5674d109-af32-4798-a8ce-0690521b463f -i td/osd-recovery-space/1/new.json 2023-11-02T09:41:07.521 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:41:07.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-space/1/new.json 2023-11-02T09:41:07.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=e9475e6c-4956-4813-8dbd-b1d4bc7eb092 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.316946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCybkNlXgqcNhAA/A2OGL8XeHekRpVCd5ro+A== --osd-uuid 5674d109-af32-4798-a8ce-0690521b463f 2023-11-02T09:41:07.562 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:41:07.560+0000 7f6f0f96e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:41:07.605 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:41:07.602+0000 7f6f0f96e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:41:07.606 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:41:07.603+0000 7f6f0f96e500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:41:07.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:41:07.604+0000 7f6f0f96e500 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2023-11-02T09:41:07.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:41:07.604+0000 7f6f0f96e500 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2023-11-02T09:41:07.607 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:41:07.604+0000 7f6f0f96e500 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2023-11-02T09:41:07.703 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:41:07.695+0000 7f6f0f96e500 -1 bluestore(td/osd-recovery-space/1) _read_fsid unparsable uuid 2023-11-02T09:41:13.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-space/1/keyring 2023-11-02T09:41:13.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-11-02T09:41:13.877 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2023-11-02T09:41:13.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-11-02T09:41:13.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-11-02T09:41:14.393 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2023-11-02T09:41:14.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-11-02T09:41:14.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=e9475e6c-4956-4813-8dbd-b1d4bc7eb092 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.316946/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-11-02T09:41:14.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-11-02T09:41:14.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-11-02T09:41:14.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-11-02T09:41:14.420 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:41:14.418+0000 7f229bb07500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:41:14.435 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:41:14.433+0000 7f229bb07500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:41:14.436 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:41:14.434+0000 7f229bb07500 -1 WARNING: all dangerous and experimental features are enabled. 2023-11-02T09:41:14.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-11-02T09:41:14.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-11-02T09:41:14.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-11-02T09:41:14.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-11-02T09:41:14.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-11-02T09:41:14.807 INFO:tasks.workunit.client.0.smithi116.stdout:0 2023-11-02T09:41:14.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:14.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-11-02T09:41:14.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:14.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:41:14.970 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:41:14.968+0000 7f229bb07500 -1 Falling back to public interface 2023-11-02T09:41:15.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:16.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:16.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:16.184 INFO:tasks.workunit.client.0.smithi116.stdout:1 2023-11-02T09:41:16.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-11-02T09:41:16.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:16.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:41:16.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:17.281 INFO:tasks.workunit.client.0.smithi116.stderr:2023-11-02T09:41:17.276+0000 7f229bb07500 -1 osd.1 0 log_to_monitors true 2023-11-02T09:41:17.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:17.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:17.628 INFO:tasks.workunit.client.0.smithi116.stdout:2 2023-11-02T09:41:17.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-11-02T09:41:17.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:17.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:41:18.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:19.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:19.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:19.059 INFO:tasks.workunit.client.0.smithi116.stdout:3 2023-11-02T09:41:19.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-11-02T09:41:19.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:19.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:41:19.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:20.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:20.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:20.436 INFO:tasks.workunit.client.0.smithi116.stdout:4 2023-11-02T09:41:20.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-11-02T09:41:20.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:20.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:41:20.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:21.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:21.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:21.921 INFO:tasks.workunit.client.0.smithi116.stdout:5 2023-11-02T09:41:21.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-11-02T09:41:21.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:21.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:41:22.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:23.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:23.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:23.417 INFO:tasks.workunit.client.0.smithi116.stdout:6 2023-11-02T09:41:23.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-11-02T09:41:23.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:23.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:41:23.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:24.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:24.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:24.802 INFO:tasks.workunit.client.0.smithi116.stdout:7 2023-11-02T09:41:24.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-11-02T09:41:24.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:24.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:41:25.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:26.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:26.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:26.288 INFO:tasks.workunit.client.0.smithi116.stdout:8 2023-11-02T09:41:26.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-11-02T09:41:26.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:26.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:41:26.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:27.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:27.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:27.700 INFO:tasks.workunit.client.0.smithi116.stdout:9 2023-11-02T09:41:27.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-11-02T09:41:27.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:27.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:41:28.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:29.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:29.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:29.116 INFO:tasks.workunit.client.0.smithi116.stdout:10 2023-11-02T09:41:29.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-11-02T09:41:29.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:29.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:41:29.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:30.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:30.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:30.529 INFO:tasks.workunit.client.0.smithi116.stdout:11 2023-11-02T09:41:30.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-11-02T09:41:30.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:30.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:41:31.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:32.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:32.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:32.062 INFO:tasks.workunit.client.0.smithi116.stdout:12 2023-11-02T09:41:32.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-11-02T09:41:32.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:32.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:41:32.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:33.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:33.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:33.870 INFO:tasks.workunit.client.0.smithi116.stdout:13 2023-11-02T09:41:33.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-11-02T09:41:33.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:33.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:41:34.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:35.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:35.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:35.620 INFO:tasks.workunit.client.0.smithi116.stdout:14 2023-11-02T09:41:35.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-11-02T09:41:35.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:35.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:41:36.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:37.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:37.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:37.456 INFO:tasks.workunit.client.0.smithi116.stdout:15 2023-11-02T09:41:37.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-11-02T09:41:37.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:37.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:41:38.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:39.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:39.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:39.515 INFO:tasks.workunit.client.0.smithi116.stdout:16 2023-11-02T09:41:39.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-11-02T09:41:39.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:39.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:41:40.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-11-02T09:41:41.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-11-02T09:41:41.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-11-02T09:41:41.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-11-02T09:41:41.203 INFO:tasks.workunit.client.0.smithi116.stdout:17 2023-11-02T09:41:41.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-11-02T09:41:41.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-11-02T09:41:41.612 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/319795,v1:127.0.0.1:6811/319795] [v2:127.0.0.1:6812/319795,v1:127.0.0.1:6813/319795] exists,up 5674d109-af32-4798-a8ce-0690521b463f 2023-11-02T09:41:41.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-11-02T09:41:41.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-11-02T09:41:41.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-11-02T09:41:41.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:102: TEST_recovery_test_simple: ceph osd set-nearfull-ratio .40 2023-11-02T09:41:42.246 INFO:tasks.workunit.client.0.smithi116.stderr:osd set-nearfull-ratio 0.4 2023-11-02T09:41:42.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:103: TEST_recovery_test_simple: ceph osd set-backfillfull-ratio .45 2023-11-02T09:41:42.999 INFO:tasks.workunit.client.0.smithi116.stderr:osd set-backfillfull-ratio 0.45 2023-11-02T09:41:43.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:104: TEST_recovery_test_simple: ceph osd set-full-ratio .50 2023-11-02T09:41:43.752 INFO:tasks.workunit.client.0.smithi116.stderr:osd set-full-ratio 0.5 2023-11-02T09:41:43.764 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:106: TEST_recovery_test_simple: seq 1 1 2023-11-02T09:41:43.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:106: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2023-11-02T09:41:43.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:108: TEST_recovery_test_simple: create_pool test1 1 1 2023-11-02T09:41:43.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2023-11-02T09:41:44.723 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test1' already exists 2023-11-02T09:41:44.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-11-02T09:41:45.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:109: TEST_recovery_test_simple: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2023-11-02T09:41:46.471 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 size to 1 2023-11-02T09:41:46.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:112: TEST_recovery_test_simple: wait_for_clean 2023-11-02T09:41:46.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-11-02T09:41:46.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-11-02T09:41:46.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-11-02T09:41:46.483 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-11-02T09:41:46.484 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T09:41:46.485 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T09:41:46.485 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T09:41:46.485 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T09:41:46.485 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T09:41:46.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2023-11-02T09:41:46.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-11-02T09:41:46.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-11-02T09:41:46.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-11-02T09:41:46.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T09:41:46.564 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T09:41:47.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T09:41:47.022 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2023-11-02T09:41:47.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T09:41:47.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:41:47.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T09:41:47.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836496 2023-11-02T09:41:47.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836496 2023-11-02T09:41:47.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836496' 2023-11-02T09:41:47.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:41:47.145 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T09:41:47.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672967 2023-11-02T09:41:47.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672967 2023-11-02T09:41:47.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836496 1-42949672967' 2023-11-02T09:41:47.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:41:47.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836496 2023-11-02T09:41:47.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:41:47.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T09:41:47.261 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836496 2023-11-02T09:41:47.261 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:41:47.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836496 2023-11-02T09:41:47.263 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836496 2023-11-02T09:41:47.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2023-11-02T09:41:47.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:41:47.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836493 -lt 21474836496 2023-11-02T09:41:47.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-11-02T09:41:48.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-11-02T09:41:48.672 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:41:49.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836496 -lt 21474836496 2023-11-02T09:41:49.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:41:49.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672967 2023-11-02T09:41:49.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:41:49.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T09:41:49.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672967 2023-11-02T09:41:49.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:41:49.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672967 2023-11-02T09:41:49.054 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672967 2023-11-02T09:41:49.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2023-11-02T09:41:49.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T09:41:49.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672967 -lt 42949672967 2023-11-02T09:41:49.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-11-02T09:41:49.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:41:49.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:41:49.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2023-11-02T09:41:49.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-11-02T09:41:49.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-11-02T09:41:49.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-11-02T09:41:49.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-11-02T09:41:49.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-11-02T09:41:49.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-11-02T09:41:49.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-11-02T09:41:50.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-11-02T09:41:50.296 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-11-02T09:41:50.297 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:41:50.297 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:41:50.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2023-11-02T09:41:50.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-11-02T09:41:50.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-11-02T09:41:50.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: dd if=/dev/urandom of=td/osd-recovery-space/datafile bs=1024 count=5 2023-11-02T09:41:50.810 INFO:tasks.workunit.client.0.smithi116.stderr:5+0 records in 2023-11-02T09:41:50.810 INFO:tasks.workunit.client.0.smithi116.stderr:5+0 records out 2023-11-02T09:41:50.811 INFO:tasks.workunit.client.0.smithi116.stderr:5120 bytes (5.1 kB, 5.0 KiB) copied, 6.5253e-05 s, 78.5 MB/s 2023-11-02T09:41:50.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: seq 1 600 2023-11-02T09:41:50.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:50.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj1 td/osd-recovery-space/datafile 2023-11-02T09:41:50.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:50.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj2 td/osd-recovery-space/datafile 2023-11-02T09:41:51.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:51.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj3 td/osd-recovery-space/datafile 2023-11-02T09:41:51.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:51.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj4 td/osd-recovery-space/datafile 2023-11-02T09:41:51.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:51.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj5 td/osd-recovery-space/datafile 2023-11-02T09:41:51.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:51.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj6 td/osd-recovery-space/datafile 2023-11-02T09:41:51.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:51.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj7 td/osd-recovery-space/datafile 2023-11-02T09:41:51.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:51.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj8 td/osd-recovery-space/datafile 2023-11-02T09:41:51.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:51.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj9 td/osd-recovery-space/datafile 2023-11-02T09:41:51.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:51.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj10 td/osd-recovery-space/datafile 2023-11-02T09:41:51.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:51.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj11 td/osd-recovery-space/datafile 2023-11-02T09:41:51.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:51.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj12 td/osd-recovery-space/datafile 2023-11-02T09:41:51.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:51.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj13 td/osd-recovery-space/datafile 2023-11-02T09:41:51.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:51.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj14 td/osd-recovery-space/datafile 2023-11-02T09:41:51.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:51.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj15 td/osd-recovery-space/datafile 2023-11-02T09:41:52.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:52.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj16 td/osd-recovery-space/datafile 2023-11-02T09:41:52.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:52.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj17 td/osd-recovery-space/datafile 2023-11-02T09:41:52.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:52.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj18 td/osd-recovery-space/datafile 2023-11-02T09:41:52.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:52.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj19 td/osd-recovery-space/datafile 2023-11-02T09:41:52.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:52.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj20 td/osd-recovery-space/datafile 2023-11-02T09:41:52.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:52.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj21 td/osd-recovery-space/datafile 2023-11-02T09:41:52.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:52.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj22 td/osd-recovery-space/datafile 2023-11-02T09:41:52.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:52.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj23 td/osd-recovery-space/datafile 2023-11-02T09:41:52.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:52.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj24 td/osd-recovery-space/datafile 2023-11-02T09:41:52.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:52.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj25 td/osd-recovery-space/datafile 2023-11-02T09:41:52.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:52.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj26 td/osd-recovery-space/datafile 2023-11-02T09:41:53.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:53.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj27 td/osd-recovery-space/datafile 2023-11-02T09:41:53.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:53.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj28 td/osd-recovery-space/datafile 2023-11-02T09:41:53.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:53.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj29 td/osd-recovery-space/datafile 2023-11-02T09:41:53.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:53.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj30 td/osd-recovery-space/datafile 2023-11-02T09:41:53.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:53.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj31 td/osd-recovery-space/datafile 2023-11-02T09:41:53.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:53.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj32 td/osd-recovery-space/datafile 2023-11-02T09:41:53.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:53.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj33 td/osd-recovery-space/datafile 2023-11-02T09:41:53.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:53.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj34 td/osd-recovery-space/datafile 2023-11-02T09:41:53.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:53.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj35 td/osd-recovery-space/datafile 2023-11-02T09:41:53.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:53.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj36 td/osd-recovery-space/datafile 2023-11-02T09:41:53.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:53.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj37 td/osd-recovery-space/datafile 2023-11-02T09:41:53.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:53.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj38 td/osd-recovery-space/datafile 2023-11-02T09:41:53.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:53.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj39 td/osd-recovery-space/datafile 2023-11-02T09:41:54.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:54.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj40 td/osd-recovery-space/datafile 2023-11-02T09:41:54.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:54.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj41 td/osd-recovery-space/datafile 2023-11-02T09:41:54.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:54.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj42 td/osd-recovery-space/datafile 2023-11-02T09:41:54.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:54.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj43 td/osd-recovery-space/datafile 2023-11-02T09:41:54.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:54.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj44 td/osd-recovery-space/datafile 2023-11-02T09:41:54.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:54.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj45 td/osd-recovery-space/datafile 2023-11-02T09:41:54.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:54.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj46 td/osd-recovery-space/datafile 2023-11-02T09:41:54.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:54.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj47 td/osd-recovery-space/datafile 2023-11-02T09:41:54.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:54.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj48 td/osd-recovery-space/datafile 2023-11-02T09:41:54.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:54.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj49 td/osd-recovery-space/datafile 2023-11-02T09:41:54.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:54.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj50 td/osd-recovery-space/datafile 2023-11-02T09:41:55.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:55.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj51 td/osd-recovery-space/datafile 2023-11-02T09:41:55.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:55.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj52 td/osd-recovery-space/datafile 2023-11-02T09:41:55.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:55.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj53 td/osd-recovery-space/datafile 2023-11-02T09:41:55.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:55.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj54 td/osd-recovery-space/datafile 2023-11-02T09:41:55.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:55.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj55 td/osd-recovery-space/datafile 2023-11-02T09:41:55.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:55.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj56 td/osd-recovery-space/datafile 2023-11-02T09:41:55.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:55.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj57 td/osd-recovery-space/datafile 2023-11-02T09:41:55.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:55.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj58 td/osd-recovery-space/datafile 2023-11-02T09:41:55.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:55.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj59 td/osd-recovery-space/datafile 2023-11-02T09:41:55.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:55.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj60 td/osd-recovery-space/datafile 2023-11-02T09:41:55.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:55.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj61 td/osd-recovery-space/datafile 2023-11-02T09:41:55.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:55.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj62 td/osd-recovery-space/datafile 2023-11-02T09:41:55.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:55.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj63 td/osd-recovery-space/datafile 2023-11-02T09:41:56.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:56.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj64 td/osd-recovery-space/datafile 2023-11-02T09:41:56.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:56.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj65 td/osd-recovery-space/datafile 2023-11-02T09:41:56.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:56.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj66 td/osd-recovery-space/datafile 2023-11-02T09:41:56.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:56.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj67 td/osd-recovery-space/datafile 2023-11-02T09:41:56.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:56.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj68 td/osd-recovery-space/datafile 2023-11-02T09:41:56.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:56.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj69 td/osd-recovery-space/datafile 2023-11-02T09:41:56.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:56.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj70 td/osd-recovery-space/datafile 2023-11-02T09:41:56.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:56.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj71 td/osd-recovery-space/datafile 2023-11-02T09:41:56.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:56.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj72 td/osd-recovery-space/datafile 2023-11-02T09:41:56.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:56.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj73 td/osd-recovery-space/datafile 2023-11-02T09:41:56.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:56.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj74 td/osd-recovery-space/datafile 2023-11-02T09:41:56.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:56.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj75 td/osd-recovery-space/datafile 2023-11-02T09:41:57.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:57.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj76 td/osd-recovery-space/datafile 2023-11-02T09:41:57.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:57.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj77 td/osd-recovery-space/datafile 2023-11-02T09:41:57.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:57.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj78 td/osd-recovery-space/datafile 2023-11-02T09:41:57.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:57.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj79 td/osd-recovery-space/datafile 2023-11-02T09:41:57.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:57.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj80 td/osd-recovery-space/datafile 2023-11-02T09:41:57.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:57.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj81 td/osd-recovery-space/datafile 2023-11-02T09:41:57.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:57.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj82 td/osd-recovery-space/datafile 2023-11-02T09:41:57.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:57.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj83 td/osd-recovery-space/datafile 2023-11-02T09:41:57.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:57.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj84 td/osd-recovery-space/datafile 2023-11-02T09:41:57.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:57.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj85 td/osd-recovery-space/datafile 2023-11-02T09:41:57.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:57.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj86 td/osd-recovery-space/datafile 2023-11-02T09:41:57.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:57.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj87 td/osd-recovery-space/datafile 2023-11-02T09:41:58.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:58.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj88 td/osd-recovery-space/datafile 2023-11-02T09:41:58.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:58.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj89 td/osd-recovery-space/datafile 2023-11-02T09:41:58.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:58.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj90 td/osd-recovery-space/datafile 2023-11-02T09:41:58.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:58.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj91 td/osd-recovery-space/datafile 2023-11-02T09:41:58.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:58.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj92 td/osd-recovery-space/datafile 2023-11-02T09:41:58.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:58.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj93 td/osd-recovery-space/datafile 2023-11-02T09:41:58.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:58.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj94 td/osd-recovery-space/datafile 2023-11-02T09:41:58.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:58.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj95 td/osd-recovery-space/datafile 2023-11-02T09:41:58.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:58.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj96 td/osd-recovery-space/datafile 2023-11-02T09:41:58.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:58.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj97 td/osd-recovery-space/datafile 2023-11-02T09:41:58.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:58.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj98 td/osd-recovery-space/datafile 2023-11-02T09:41:59.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:59.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj99 td/osd-recovery-space/datafile 2023-11-02T09:41:59.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:59.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj100 td/osd-recovery-space/datafile 2023-11-02T09:41:59.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:59.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj101 td/osd-recovery-space/datafile 2023-11-02T09:41:59.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:59.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj102 td/osd-recovery-space/datafile 2023-11-02T09:41:59.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:59.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj103 td/osd-recovery-space/datafile 2023-11-02T09:41:59.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:59.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj104 td/osd-recovery-space/datafile 2023-11-02T09:41:59.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:59.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj105 td/osd-recovery-space/datafile 2023-11-02T09:41:59.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:59.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj106 td/osd-recovery-space/datafile 2023-11-02T09:41:59.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:59.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj107 td/osd-recovery-space/datafile 2023-11-02T09:41:59.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:59.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj108 td/osd-recovery-space/datafile 2023-11-02T09:41:59.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:59.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj109 td/osd-recovery-space/datafile 2023-11-02T09:41:59.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:59.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj110 td/osd-recovery-space/datafile 2023-11-02T09:41:59.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:41:59.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj111 td/osd-recovery-space/datafile 2023-11-02T09:42:00.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:00.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj112 td/osd-recovery-space/datafile 2023-11-02T09:42:00.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:00.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj113 td/osd-recovery-space/datafile 2023-11-02T09:42:00.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:00.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj114 td/osd-recovery-space/datafile 2023-11-02T09:42:00.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:00.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj115 td/osd-recovery-space/datafile 2023-11-02T09:42:00.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:00.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj116 td/osd-recovery-space/datafile 2023-11-02T09:42:00.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:00.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj117 td/osd-recovery-space/datafile 2023-11-02T09:42:00.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:00.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj118 td/osd-recovery-space/datafile 2023-11-02T09:42:00.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:00.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj119 td/osd-recovery-space/datafile 2023-11-02T09:42:00.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:00.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj120 td/osd-recovery-space/datafile 2023-11-02T09:42:00.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:00.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj121 td/osd-recovery-space/datafile 2023-11-02T09:42:00.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:00.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj122 td/osd-recovery-space/datafile 2023-11-02T09:42:00.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:00.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj123 td/osd-recovery-space/datafile 2023-11-02T09:42:01.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:01.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj124 td/osd-recovery-space/datafile 2023-11-02T09:42:01.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:01.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj125 td/osd-recovery-space/datafile 2023-11-02T09:42:01.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:01.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj126 td/osd-recovery-space/datafile 2023-11-02T09:42:01.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:01.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj127 td/osd-recovery-space/datafile 2023-11-02T09:42:01.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:01.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj128 td/osd-recovery-space/datafile 2023-11-02T09:42:01.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:01.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj129 td/osd-recovery-space/datafile 2023-11-02T09:42:01.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:01.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj130 td/osd-recovery-space/datafile 2023-11-02T09:42:01.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:01.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj131 td/osd-recovery-space/datafile 2023-11-02T09:42:01.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:01.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj132 td/osd-recovery-space/datafile 2023-11-02T09:42:01.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:01.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj133 td/osd-recovery-space/datafile 2023-11-02T09:42:01.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:01.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj134 td/osd-recovery-space/datafile 2023-11-02T09:42:01.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:01.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj135 td/osd-recovery-space/datafile 2023-11-02T09:42:02.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:02.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj136 td/osd-recovery-space/datafile 2023-11-02T09:42:02.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:02.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj137 td/osd-recovery-space/datafile 2023-11-02T09:42:02.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:02.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj138 td/osd-recovery-space/datafile 2023-11-02T09:42:02.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:02.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj139 td/osd-recovery-space/datafile 2023-11-02T09:42:02.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:02.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj140 td/osd-recovery-space/datafile 2023-11-02T09:42:02.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:02.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj141 td/osd-recovery-space/datafile 2023-11-02T09:42:02.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:02.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj142 td/osd-recovery-space/datafile 2023-11-02T09:42:02.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:02.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj143 td/osd-recovery-space/datafile 2023-11-02T09:42:02.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:02.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj144 td/osd-recovery-space/datafile 2023-11-02T09:42:02.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:02.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj145 td/osd-recovery-space/datafile 2023-11-02T09:42:02.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:02.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj146 td/osd-recovery-space/datafile 2023-11-02T09:42:02.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:02.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj147 td/osd-recovery-space/datafile 2023-11-02T09:42:03.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:03.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj148 td/osd-recovery-space/datafile 2023-11-02T09:42:03.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:03.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj149 td/osd-recovery-space/datafile 2023-11-02T09:42:03.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:03.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj150 td/osd-recovery-space/datafile 2023-11-02T09:42:03.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:03.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj151 td/osd-recovery-space/datafile 2023-11-02T09:42:03.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:03.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj152 td/osd-recovery-space/datafile 2023-11-02T09:42:03.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:03.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj153 td/osd-recovery-space/datafile 2023-11-02T09:42:03.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:03.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj154 td/osd-recovery-space/datafile 2023-11-02T09:42:03.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:03.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj155 td/osd-recovery-space/datafile 2023-11-02T09:42:03.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:03.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj156 td/osd-recovery-space/datafile 2023-11-02T09:42:03.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:03.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj157 td/osd-recovery-space/datafile 2023-11-02T09:42:03.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:03.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj158 td/osd-recovery-space/datafile 2023-11-02T09:42:03.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:03.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj159 td/osd-recovery-space/datafile 2023-11-02T09:42:04.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:04.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj160 td/osd-recovery-space/datafile 2023-11-02T09:42:04.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:04.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj161 td/osd-recovery-space/datafile 2023-11-02T09:42:04.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:04.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj162 td/osd-recovery-space/datafile 2023-11-02T09:42:04.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:04.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj163 td/osd-recovery-space/datafile 2023-11-02T09:42:04.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:04.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj164 td/osd-recovery-space/datafile 2023-11-02T09:42:04.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:04.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj165 td/osd-recovery-space/datafile 2023-11-02T09:42:04.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:04.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj166 td/osd-recovery-space/datafile 2023-11-02T09:42:04.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:04.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj167 td/osd-recovery-space/datafile 2023-11-02T09:42:04.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:04.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj168 td/osd-recovery-space/datafile 2023-11-02T09:42:04.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:04.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj169 td/osd-recovery-space/datafile 2023-11-02T09:42:04.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:04.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj170 td/osd-recovery-space/datafile 2023-11-02T09:42:04.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:04.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj171 td/osd-recovery-space/datafile 2023-11-02T09:42:05.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:05.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj172 td/osd-recovery-space/datafile 2023-11-02T09:42:05.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:05.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj173 td/osd-recovery-space/datafile 2023-11-02T09:42:05.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:05.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj174 td/osd-recovery-space/datafile 2023-11-02T09:42:05.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:05.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj175 td/osd-recovery-space/datafile 2023-11-02T09:42:05.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:05.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj176 td/osd-recovery-space/datafile 2023-11-02T09:42:05.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:05.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj177 td/osd-recovery-space/datafile 2023-11-02T09:42:05.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:05.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj178 td/osd-recovery-space/datafile 2023-11-02T09:42:05.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:05.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj179 td/osd-recovery-space/datafile 2023-11-02T09:42:05.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:05.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj180 td/osd-recovery-space/datafile 2023-11-02T09:42:05.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:05.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj181 td/osd-recovery-space/datafile 2023-11-02T09:42:05.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:05.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj182 td/osd-recovery-space/datafile 2023-11-02T09:42:05.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:05.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj183 td/osd-recovery-space/datafile 2023-11-02T09:42:06.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:06.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj184 td/osd-recovery-space/datafile 2023-11-02T09:42:06.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:06.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj185 td/osd-recovery-space/datafile 2023-11-02T09:42:06.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:06.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj186 td/osd-recovery-space/datafile 2023-11-02T09:42:06.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:06.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj187 td/osd-recovery-space/datafile 2023-11-02T09:42:06.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:06.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj188 td/osd-recovery-space/datafile 2023-11-02T09:42:06.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:06.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj189 td/osd-recovery-space/datafile 2023-11-02T09:42:06.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:06.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj190 td/osd-recovery-space/datafile 2023-11-02T09:42:06.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:06.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj191 td/osd-recovery-space/datafile 2023-11-02T09:42:06.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:06.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj192 td/osd-recovery-space/datafile 2023-11-02T09:42:06.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:06.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj193 td/osd-recovery-space/datafile 2023-11-02T09:42:06.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:06.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj194 td/osd-recovery-space/datafile 2023-11-02T09:42:06.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:06.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj195 td/osd-recovery-space/datafile 2023-11-02T09:42:07.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:07.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj196 td/osd-recovery-space/datafile 2023-11-02T09:42:07.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:07.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj197 td/osd-recovery-space/datafile 2023-11-02T09:42:07.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:07.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj198 td/osd-recovery-space/datafile 2023-11-02T09:42:07.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:07.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj199 td/osd-recovery-space/datafile 2023-11-02T09:42:07.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:07.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj200 td/osd-recovery-space/datafile 2023-11-02T09:42:07.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:07.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj201 td/osd-recovery-space/datafile 2023-11-02T09:42:07.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:07.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj202 td/osd-recovery-space/datafile 2023-11-02T09:42:07.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:07.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj203 td/osd-recovery-space/datafile 2023-11-02T09:42:07.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:07.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj204 td/osd-recovery-space/datafile 2023-11-02T09:42:08.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:08.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj205 td/osd-recovery-space/datafile 2023-11-02T09:42:08.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:08.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj206 td/osd-recovery-space/datafile 2023-11-02T09:42:08.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:08.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj207 td/osd-recovery-space/datafile 2023-11-02T09:42:08.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:08.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj208 td/osd-recovery-space/datafile 2023-11-02T09:42:08.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:08.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj209 td/osd-recovery-space/datafile 2023-11-02T09:42:08.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:08.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj210 td/osd-recovery-space/datafile 2023-11-02T09:42:08.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:08.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj211 td/osd-recovery-space/datafile 2023-11-02T09:42:08.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:08.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj212 td/osd-recovery-space/datafile 2023-11-02T09:42:08.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:08.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj213 td/osd-recovery-space/datafile 2023-11-02T09:42:08.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:08.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj214 td/osd-recovery-space/datafile 2023-11-02T09:42:08.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:08.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj215 td/osd-recovery-space/datafile 2023-11-02T09:42:09.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:09.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj216 td/osd-recovery-space/datafile 2023-11-02T09:42:09.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:09.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj217 td/osd-recovery-space/datafile 2023-11-02T09:42:09.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:09.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj218 td/osd-recovery-space/datafile 2023-11-02T09:42:09.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:09.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj219 td/osd-recovery-space/datafile 2023-11-02T09:42:09.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:09.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj220 td/osd-recovery-space/datafile 2023-11-02T09:42:09.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:09.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj221 td/osd-recovery-space/datafile 2023-11-02T09:42:09.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:09.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj222 td/osd-recovery-space/datafile 2023-11-02T09:42:09.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:09.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj223 td/osd-recovery-space/datafile 2023-11-02T09:42:09.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:09.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj224 td/osd-recovery-space/datafile 2023-11-02T09:42:09.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:09.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj225 td/osd-recovery-space/datafile 2023-11-02T09:42:09.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:09.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj226 td/osd-recovery-space/datafile 2023-11-02T09:42:09.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:09.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj227 td/osd-recovery-space/datafile 2023-11-02T09:42:09.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:09.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj228 td/osd-recovery-space/datafile 2023-11-02T09:42:10.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:10.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj229 td/osd-recovery-space/datafile 2023-11-02T09:42:10.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:10.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj230 td/osd-recovery-space/datafile 2023-11-02T09:42:10.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:10.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj231 td/osd-recovery-space/datafile 2023-11-02T09:42:10.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:10.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj232 td/osd-recovery-space/datafile 2023-11-02T09:42:10.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:10.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj233 td/osd-recovery-space/datafile 2023-11-02T09:42:10.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:10.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj234 td/osd-recovery-space/datafile 2023-11-02T09:42:10.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:10.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj235 td/osd-recovery-space/datafile 2023-11-02T09:42:10.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:10.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj236 td/osd-recovery-space/datafile 2023-11-02T09:42:10.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:10.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj237 td/osd-recovery-space/datafile 2023-11-02T09:42:10.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:10.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj238 td/osd-recovery-space/datafile 2023-11-02T09:42:10.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:10.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj239 td/osd-recovery-space/datafile 2023-11-02T09:42:10.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:10.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj240 td/osd-recovery-space/datafile 2023-11-02T09:42:11.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:11.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj241 td/osd-recovery-space/datafile 2023-11-02T09:42:11.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:11.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj242 td/osd-recovery-space/datafile 2023-11-02T09:42:11.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:11.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj243 td/osd-recovery-space/datafile 2023-11-02T09:42:11.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:11.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj244 td/osd-recovery-space/datafile 2023-11-02T09:42:11.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:11.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj245 td/osd-recovery-space/datafile 2023-11-02T09:42:11.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:11.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj246 td/osd-recovery-space/datafile 2023-11-02T09:42:11.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:11.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj247 td/osd-recovery-space/datafile 2023-11-02T09:42:11.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:11.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj248 td/osd-recovery-space/datafile 2023-11-02T09:42:11.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:11.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj249 td/osd-recovery-space/datafile 2023-11-02T09:42:11.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:11.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj250 td/osd-recovery-space/datafile 2023-11-02T09:42:11.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:11.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj251 td/osd-recovery-space/datafile 2023-11-02T09:42:11.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:11.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj252 td/osd-recovery-space/datafile 2023-11-02T09:42:12.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:12.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj253 td/osd-recovery-space/datafile 2023-11-02T09:42:12.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:12.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj254 td/osd-recovery-space/datafile 2023-11-02T09:42:12.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:12.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj255 td/osd-recovery-space/datafile 2023-11-02T09:42:12.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:12.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj256 td/osd-recovery-space/datafile 2023-11-02T09:42:12.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:12.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj257 td/osd-recovery-space/datafile 2023-11-02T09:42:12.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:12.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj258 td/osd-recovery-space/datafile 2023-11-02T09:42:12.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:12.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj259 td/osd-recovery-space/datafile 2023-11-02T09:42:12.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:12.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj260 td/osd-recovery-space/datafile 2023-11-02T09:42:12.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:12.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj261 td/osd-recovery-space/datafile 2023-11-02T09:42:12.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:12.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj262 td/osd-recovery-space/datafile 2023-11-02T09:42:12.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:12.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj263 td/osd-recovery-space/datafile 2023-11-02T09:42:12.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:12.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj264 td/osd-recovery-space/datafile 2023-11-02T09:42:13.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:13.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj265 td/osd-recovery-space/datafile 2023-11-02T09:42:13.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:13.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj266 td/osd-recovery-space/datafile 2023-11-02T09:42:13.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:13.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj267 td/osd-recovery-space/datafile 2023-11-02T09:42:13.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:13.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj268 td/osd-recovery-space/datafile 2023-11-02T09:42:13.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:13.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj269 td/osd-recovery-space/datafile 2023-11-02T09:42:13.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:13.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj270 td/osd-recovery-space/datafile 2023-11-02T09:42:13.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:13.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj271 td/osd-recovery-space/datafile 2023-11-02T09:42:13.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:13.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj272 td/osd-recovery-space/datafile 2023-11-02T09:42:13.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:13.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj273 td/osd-recovery-space/datafile 2023-11-02T09:42:13.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:13.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj274 td/osd-recovery-space/datafile 2023-11-02T09:42:13.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:13.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj275 td/osd-recovery-space/datafile 2023-11-02T09:42:14.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:14.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj276 td/osd-recovery-space/datafile 2023-11-02T09:42:14.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:14.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj277 td/osd-recovery-space/datafile 2023-11-02T09:42:14.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:14.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj278 td/osd-recovery-space/datafile 2023-11-02T09:42:14.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:14.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj279 td/osd-recovery-space/datafile 2023-11-02T09:42:14.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:14.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj280 td/osd-recovery-space/datafile 2023-11-02T09:42:14.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:14.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj281 td/osd-recovery-space/datafile 2023-11-02T09:42:14.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:14.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj282 td/osd-recovery-space/datafile 2023-11-02T09:42:14.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:14.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj283 td/osd-recovery-space/datafile 2023-11-02T09:42:14.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:14.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj284 td/osd-recovery-space/datafile 2023-11-02T09:42:14.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:14.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj285 td/osd-recovery-space/datafile 2023-11-02T09:42:14.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:14.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj286 td/osd-recovery-space/datafile 2023-11-02T09:42:14.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:14.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj287 td/osd-recovery-space/datafile 2023-11-02T09:42:15.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:15.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj288 td/osd-recovery-space/datafile 2023-11-02T09:42:15.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:15.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj289 td/osd-recovery-space/datafile 2023-11-02T09:42:15.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:15.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj290 td/osd-recovery-space/datafile 2023-11-02T09:42:15.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:15.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj291 td/osd-recovery-space/datafile 2023-11-02T09:42:15.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:15.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj292 td/osd-recovery-space/datafile 2023-11-02T09:42:15.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:15.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj293 td/osd-recovery-space/datafile 2023-11-02T09:42:15.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:15.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj294 td/osd-recovery-space/datafile 2023-11-02T09:42:15.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:15.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj295 td/osd-recovery-space/datafile 2023-11-02T09:42:15.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:15.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj296 td/osd-recovery-space/datafile 2023-11-02T09:42:15.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:15.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj297 td/osd-recovery-space/datafile 2023-11-02T09:42:15.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:15.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj298 td/osd-recovery-space/datafile 2023-11-02T09:42:15.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:15.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj299 td/osd-recovery-space/datafile 2023-11-02T09:42:16.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:16.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj300 td/osd-recovery-space/datafile 2023-11-02T09:42:16.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:16.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj301 td/osd-recovery-space/datafile 2023-11-02T09:42:16.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:16.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj302 td/osd-recovery-space/datafile 2023-11-02T09:42:16.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:16.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj303 td/osd-recovery-space/datafile 2023-11-02T09:42:16.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:16.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj304 td/osd-recovery-space/datafile 2023-11-02T09:42:16.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:16.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj305 td/osd-recovery-space/datafile 2023-11-02T09:42:16.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:16.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj306 td/osd-recovery-space/datafile 2023-11-02T09:42:16.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:16.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj307 td/osd-recovery-space/datafile 2023-11-02T09:42:16.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:16.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj308 td/osd-recovery-space/datafile 2023-11-02T09:42:16.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:16.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj309 td/osd-recovery-space/datafile 2023-11-02T09:42:16.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:16.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj310 td/osd-recovery-space/datafile 2023-11-02T09:42:16.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:16.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj311 td/osd-recovery-space/datafile 2023-11-02T09:42:17.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:17.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj312 td/osd-recovery-space/datafile 2023-11-02T09:42:17.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:17.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj313 td/osd-recovery-space/datafile 2023-11-02T09:42:17.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:17.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj314 td/osd-recovery-space/datafile 2023-11-02T09:42:17.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:17.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj315 td/osd-recovery-space/datafile 2023-11-02T09:42:17.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:17.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj316 td/osd-recovery-space/datafile 2023-11-02T09:42:17.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:17.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj317 td/osd-recovery-space/datafile 2023-11-02T09:42:17.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:17.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj318 td/osd-recovery-space/datafile 2023-11-02T09:42:17.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:17.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj319 td/osd-recovery-space/datafile 2023-11-02T09:42:17.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:17.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj320 td/osd-recovery-space/datafile 2023-11-02T09:42:17.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:17.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj321 td/osd-recovery-space/datafile 2023-11-02T09:42:17.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:17.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj322 td/osd-recovery-space/datafile 2023-11-02T09:42:17.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:17.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj323 td/osd-recovery-space/datafile 2023-11-02T09:42:18.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:18.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj324 td/osd-recovery-space/datafile 2023-11-02T09:42:18.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:18.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj325 td/osd-recovery-space/datafile 2023-11-02T09:42:18.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:18.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj326 td/osd-recovery-space/datafile 2023-11-02T09:42:18.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:18.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj327 td/osd-recovery-space/datafile 2023-11-02T09:42:18.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:18.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj328 td/osd-recovery-space/datafile 2023-11-02T09:42:18.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:18.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj329 td/osd-recovery-space/datafile 2023-11-02T09:42:18.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:18.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj330 td/osd-recovery-space/datafile 2023-11-02T09:42:18.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:18.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj331 td/osd-recovery-space/datafile 2023-11-02T09:42:18.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:18.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj332 td/osd-recovery-space/datafile 2023-11-02T09:42:18.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:18.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj333 td/osd-recovery-space/datafile 2023-11-02T09:42:18.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:18.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj334 td/osd-recovery-space/datafile 2023-11-02T09:42:18.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:18.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj335 td/osd-recovery-space/datafile 2023-11-02T09:42:18.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:18.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj336 td/osd-recovery-space/datafile 2023-11-02T09:42:19.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:19.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj337 td/osd-recovery-space/datafile 2023-11-02T09:42:19.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:19.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj338 td/osd-recovery-space/datafile 2023-11-02T09:42:19.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:19.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj339 td/osd-recovery-space/datafile 2023-11-02T09:42:19.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:19.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj340 td/osd-recovery-space/datafile 2023-11-02T09:42:19.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:19.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj341 td/osd-recovery-space/datafile 2023-11-02T09:42:19.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:19.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj342 td/osd-recovery-space/datafile 2023-11-02T09:42:19.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:19.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj343 td/osd-recovery-space/datafile 2023-11-02T09:42:19.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:19.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj344 td/osd-recovery-space/datafile 2023-11-02T09:42:19.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:19.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj345 td/osd-recovery-space/datafile 2023-11-02T09:42:19.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:19.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj346 td/osd-recovery-space/datafile 2023-11-02T09:42:19.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:19.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj347 td/osd-recovery-space/datafile 2023-11-02T09:42:19.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:19.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj348 td/osd-recovery-space/datafile 2023-11-02T09:42:19.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:19.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj349 td/osd-recovery-space/datafile 2023-11-02T09:42:19.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:19.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj350 td/osd-recovery-space/datafile 2023-11-02T09:42:19.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:19.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj351 td/osd-recovery-space/datafile 2023-11-02T09:42:20.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:20.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj352 td/osd-recovery-space/datafile 2023-11-02T09:42:20.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:20.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj353 td/osd-recovery-space/datafile 2023-11-02T09:42:20.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:20.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj354 td/osd-recovery-space/datafile 2023-11-02T09:42:20.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:20.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj355 td/osd-recovery-space/datafile 2023-11-02T09:42:20.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:20.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj356 td/osd-recovery-space/datafile 2023-11-02T09:42:20.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:20.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj357 td/osd-recovery-space/datafile 2023-11-02T09:42:20.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:20.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj358 td/osd-recovery-space/datafile 2023-11-02T09:42:20.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:20.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj359 td/osd-recovery-space/datafile 2023-11-02T09:42:20.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:20.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj360 td/osd-recovery-space/datafile 2023-11-02T09:42:20.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:20.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj361 td/osd-recovery-space/datafile 2023-11-02T09:42:20.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:20.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj362 td/osd-recovery-space/datafile 2023-11-02T09:42:20.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:20.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj363 td/osd-recovery-space/datafile 2023-11-02T09:42:20.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:20.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj364 td/osd-recovery-space/datafile 2023-11-02T09:42:20.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:20.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj365 td/osd-recovery-space/datafile 2023-11-02T09:42:21.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:21.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj366 td/osd-recovery-space/datafile 2023-11-02T09:42:21.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:21.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj367 td/osd-recovery-space/datafile 2023-11-02T09:42:21.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:21.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj368 td/osd-recovery-space/datafile 2023-11-02T09:42:21.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:21.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj369 td/osd-recovery-space/datafile 2023-11-02T09:42:21.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:21.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj370 td/osd-recovery-space/datafile 2023-11-02T09:42:21.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:21.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj371 td/osd-recovery-space/datafile 2023-11-02T09:42:21.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:21.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj372 td/osd-recovery-space/datafile 2023-11-02T09:42:21.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:21.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj373 td/osd-recovery-space/datafile 2023-11-02T09:42:21.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:21.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj374 td/osd-recovery-space/datafile 2023-11-02T09:42:21.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:21.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj375 td/osd-recovery-space/datafile 2023-11-02T09:42:21.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:21.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj376 td/osd-recovery-space/datafile 2023-11-02T09:42:21.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:21.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj377 td/osd-recovery-space/datafile 2023-11-02T09:42:21.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:21.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj378 td/osd-recovery-space/datafile 2023-11-02T09:42:21.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:21.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj379 td/osd-recovery-space/datafile 2023-11-02T09:42:21.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:21.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj380 td/osd-recovery-space/datafile 2023-11-02T09:42:22.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:22.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj381 td/osd-recovery-space/datafile 2023-11-02T09:42:22.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:22.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj382 td/osd-recovery-space/datafile 2023-11-02T09:42:22.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:22.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj383 td/osd-recovery-space/datafile 2023-11-02T09:42:22.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:22.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj384 td/osd-recovery-space/datafile 2023-11-02T09:42:22.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:22.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj385 td/osd-recovery-space/datafile 2023-11-02T09:42:22.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:22.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj386 td/osd-recovery-space/datafile 2023-11-02T09:42:22.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:22.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj387 td/osd-recovery-space/datafile 2023-11-02T09:42:22.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:22.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj388 td/osd-recovery-space/datafile 2023-11-02T09:42:22.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:22.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj389 td/osd-recovery-space/datafile 2023-11-02T09:42:22.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:22.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj390 td/osd-recovery-space/datafile 2023-11-02T09:42:22.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:22.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj391 td/osd-recovery-space/datafile 2023-11-02T09:42:22.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:22.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj392 td/osd-recovery-space/datafile 2023-11-02T09:42:22.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:22.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj393 td/osd-recovery-space/datafile 2023-11-02T09:42:22.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:22.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj394 td/osd-recovery-space/datafile 2023-11-02T09:42:23.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:23.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj395 td/osd-recovery-space/datafile 2023-11-02T09:42:23.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:23.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj396 td/osd-recovery-space/datafile 2023-11-02T09:42:23.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:23.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj397 td/osd-recovery-space/datafile 2023-11-02T09:42:23.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:23.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj398 td/osd-recovery-space/datafile 2023-11-02T09:42:23.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:23.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj399 td/osd-recovery-space/datafile 2023-11-02T09:42:23.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:23.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj400 td/osd-recovery-space/datafile 2023-11-02T09:42:23.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:23.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj401 td/osd-recovery-space/datafile 2023-11-02T09:42:23.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:23.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj402 td/osd-recovery-space/datafile 2023-11-02T09:42:23.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:23.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj403 td/osd-recovery-space/datafile 2023-11-02T09:42:23.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:23.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj404 td/osd-recovery-space/datafile 2023-11-02T09:42:23.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:23.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj405 td/osd-recovery-space/datafile 2023-11-02T09:42:23.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:23.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj406 td/osd-recovery-space/datafile 2023-11-02T09:42:23.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:23.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj407 td/osd-recovery-space/datafile 2023-11-02T09:42:23.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:23.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj408 td/osd-recovery-space/datafile 2023-11-02T09:42:24.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:24.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj409 td/osd-recovery-space/datafile 2023-11-02T09:42:24.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:24.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj410 td/osd-recovery-space/datafile 2023-11-02T09:42:24.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:24.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj411 td/osd-recovery-space/datafile 2023-11-02T09:42:24.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:24.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj412 td/osd-recovery-space/datafile 2023-11-02T09:42:24.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:24.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj413 td/osd-recovery-space/datafile 2023-11-02T09:42:24.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:24.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj414 td/osd-recovery-space/datafile 2023-11-02T09:42:24.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:24.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj415 td/osd-recovery-space/datafile 2023-11-02T09:42:24.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:24.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj416 td/osd-recovery-space/datafile 2023-11-02T09:42:24.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:24.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj417 td/osd-recovery-space/datafile 2023-11-02T09:42:24.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:24.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj418 td/osd-recovery-space/datafile 2023-11-02T09:42:24.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:24.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj419 td/osd-recovery-space/datafile 2023-11-02T09:42:24.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:24.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj420 td/osd-recovery-space/datafile 2023-11-02T09:42:24.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:24.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj421 td/osd-recovery-space/datafile 2023-11-02T09:42:25.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:25.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj422 td/osd-recovery-space/datafile 2023-11-02T09:42:25.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:25.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj423 td/osd-recovery-space/datafile 2023-11-02T09:42:25.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:25.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj424 td/osd-recovery-space/datafile 2023-11-02T09:42:25.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:25.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj425 td/osd-recovery-space/datafile 2023-11-02T09:42:25.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:25.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj426 td/osd-recovery-space/datafile 2023-11-02T09:42:25.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:25.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj427 td/osd-recovery-space/datafile 2023-11-02T09:42:25.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:25.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj428 td/osd-recovery-space/datafile 2023-11-02T09:42:25.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:25.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj429 td/osd-recovery-space/datafile 2023-11-02T09:42:25.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:25.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj430 td/osd-recovery-space/datafile 2023-11-02T09:42:25.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:25.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj431 td/osd-recovery-space/datafile 2023-11-02T09:42:25.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:25.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj432 td/osd-recovery-space/datafile 2023-11-02T09:42:25.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:25.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj433 td/osd-recovery-space/datafile 2023-11-02T09:42:25.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:25.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj434 td/osd-recovery-space/datafile 2023-11-02T09:42:25.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:25.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj435 td/osd-recovery-space/datafile 2023-11-02T09:42:25.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:25.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj436 td/osd-recovery-space/datafile 2023-11-02T09:42:26.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:26.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj437 td/osd-recovery-space/datafile 2023-11-02T09:42:26.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:26.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj438 td/osd-recovery-space/datafile 2023-11-02T09:42:26.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:26.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj439 td/osd-recovery-space/datafile 2023-11-02T09:42:26.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:26.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj440 td/osd-recovery-space/datafile 2023-11-02T09:42:26.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:26.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj441 td/osd-recovery-space/datafile 2023-11-02T09:42:26.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:26.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj442 td/osd-recovery-space/datafile 2023-11-02T09:42:26.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:26.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj443 td/osd-recovery-space/datafile 2023-11-02T09:42:26.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:26.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj444 td/osd-recovery-space/datafile 2023-11-02T09:42:26.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:26.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj445 td/osd-recovery-space/datafile 2023-11-02T09:42:26.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:26.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj446 td/osd-recovery-space/datafile 2023-11-02T09:42:26.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:26.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj447 td/osd-recovery-space/datafile 2023-11-02T09:42:26.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:26.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj448 td/osd-recovery-space/datafile 2023-11-02T09:42:26.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:26.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj449 td/osd-recovery-space/datafile 2023-11-02T09:42:26.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:26.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj450 td/osd-recovery-space/datafile 2023-11-02T09:42:27.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:27.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj451 td/osd-recovery-space/datafile 2023-11-02T09:42:27.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:27.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj452 td/osd-recovery-space/datafile 2023-11-02T09:42:27.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:27.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj453 td/osd-recovery-space/datafile 2023-11-02T09:42:27.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:27.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj454 td/osd-recovery-space/datafile 2023-11-02T09:42:27.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:27.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj455 td/osd-recovery-space/datafile 2023-11-02T09:42:27.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:27.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj456 td/osd-recovery-space/datafile 2023-11-02T09:42:27.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:27.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj457 td/osd-recovery-space/datafile 2023-11-02T09:42:27.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:27.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj458 td/osd-recovery-space/datafile 2023-11-02T09:42:27.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:27.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj459 td/osd-recovery-space/datafile 2023-11-02T09:42:27.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:27.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj460 td/osd-recovery-space/datafile 2023-11-02T09:42:27.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:27.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj461 td/osd-recovery-space/datafile 2023-11-02T09:42:27.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:27.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj462 td/osd-recovery-space/datafile 2023-11-02T09:42:27.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:27.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj463 td/osd-recovery-space/datafile 2023-11-02T09:42:27.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:27.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj464 td/osd-recovery-space/datafile 2023-11-02T09:42:27.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:27.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj465 td/osd-recovery-space/datafile 2023-11-02T09:42:28.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:28.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj466 td/osd-recovery-space/datafile 2023-11-02T09:42:28.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:28.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj467 td/osd-recovery-space/datafile 2023-11-02T09:42:28.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:28.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj468 td/osd-recovery-space/datafile 2023-11-02T09:42:28.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:28.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj469 td/osd-recovery-space/datafile 2023-11-02T09:42:28.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:28.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj470 td/osd-recovery-space/datafile 2023-11-02T09:42:28.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:28.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj471 td/osd-recovery-space/datafile 2023-11-02T09:42:28.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:28.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj472 td/osd-recovery-space/datafile 2023-11-02T09:42:28.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:28.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj473 td/osd-recovery-space/datafile 2023-11-02T09:42:28.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:28.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj474 td/osd-recovery-space/datafile 2023-11-02T09:42:28.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:28.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj475 td/osd-recovery-space/datafile 2023-11-02T09:42:28.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:28.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj476 td/osd-recovery-space/datafile 2023-11-02T09:42:28.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:28.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj477 td/osd-recovery-space/datafile 2023-11-02T09:42:28.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:28.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj478 td/osd-recovery-space/datafile 2023-11-02T09:42:28.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:28.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj479 td/osd-recovery-space/datafile 2023-11-02T09:42:29.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:29.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj480 td/osd-recovery-space/datafile 2023-11-02T09:42:29.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:29.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj481 td/osd-recovery-space/datafile 2023-11-02T09:42:29.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:29.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj482 td/osd-recovery-space/datafile 2023-11-02T09:42:29.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:29.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj483 td/osd-recovery-space/datafile 2023-11-02T09:42:29.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:29.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj484 td/osd-recovery-space/datafile 2023-11-02T09:42:29.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:29.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj485 td/osd-recovery-space/datafile 2023-11-02T09:42:29.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:29.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj486 td/osd-recovery-space/datafile 2023-11-02T09:42:29.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:29.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj487 td/osd-recovery-space/datafile 2023-11-02T09:42:29.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:29.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj488 td/osd-recovery-space/datafile 2023-11-02T09:42:29.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:29.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj489 td/osd-recovery-space/datafile 2023-11-02T09:42:29.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:29.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj490 td/osd-recovery-space/datafile 2023-11-02T09:42:29.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:29.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj491 td/osd-recovery-space/datafile 2023-11-02T09:42:29.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:29.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj492 td/osd-recovery-space/datafile 2023-11-02T09:42:29.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:29.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj493 td/osd-recovery-space/datafile 2023-11-02T09:42:29.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:29.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj494 td/osd-recovery-space/datafile 2023-11-02T09:42:30.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:30.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj495 td/osd-recovery-space/datafile 2023-11-02T09:42:30.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:30.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj496 td/osd-recovery-space/datafile 2023-11-02T09:42:30.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:30.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj497 td/osd-recovery-space/datafile 2023-11-02T09:42:30.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:30.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj498 td/osd-recovery-space/datafile 2023-11-02T09:42:30.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:30.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj499 td/osd-recovery-space/datafile 2023-11-02T09:42:30.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:30.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj500 td/osd-recovery-space/datafile 2023-11-02T09:42:30.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:30.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj501 td/osd-recovery-space/datafile 2023-11-02T09:42:30.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:30.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj502 td/osd-recovery-space/datafile 2023-11-02T09:42:30.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:30.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj503 td/osd-recovery-space/datafile 2023-11-02T09:42:30.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:30.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj504 td/osd-recovery-space/datafile 2023-11-02T09:42:30.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:30.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj505 td/osd-recovery-space/datafile 2023-11-02T09:42:30.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:30.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj506 td/osd-recovery-space/datafile 2023-11-02T09:42:30.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:30.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj507 td/osd-recovery-space/datafile 2023-11-02T09:42:30.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:30.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj508 td/osd-recovery-space/datafile 2023-11-02T09:42:31.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:31.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj509 td/osd-recovery-space/datafile 2023-11-02T09:42:31.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:31.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj510 td/osd-recovery-space/datafile 2023-11-02T09:42:31.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:31.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj511 td/osd-recovery-space/datafile 2023-11-02T09:42:31.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:31.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj512 td/osd-recovery-space/datafile 2023-11-02T09:42:31.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:31.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj513 td/osd-recovery-space/datafile 2023-11-02T09:42:31.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:31.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj514 td/osd-recovery-space/datafile 2023-11-02T09:42:31.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:31.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj515 td/osd-recovery-space/datafile 2023-11-02T09:42:31.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:31.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj516 td/osd-recovery-space/datafile 2023-11-02T09:42:31.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:31.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj517 td/osd-recovery-space/datafile 2023-11-02T09:42:31.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:31.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj518 td/osd-recovery-space/datafile 2023-11-02T09:42:31.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:31.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj519 td/osd-recovery-space/datafile 2023-11-02T09:42:31.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:31.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj520 td/osd-recovery-space/datafile 2023-11-02T09:42:31.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:31.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj521 td/osd-recovery-space/datafile 2023-11-02T09:42:31.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:31.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj522 td/osd-recovery-space/datafile 2023-11-02T09:42:31.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:31.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj523 td/osd-recovery-space/datafile 2023-11-02T09:42:32.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:32.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj524 td/osd-recovery-space/datafile 2023-11-02T09:42:32.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:32.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj525 td/osd-recovery-space/datafile 2023-11-02T09:42:32.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:32.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj526 td/osd-recovery-space/datafile 2023-11-02T09:42:32.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:32.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj527 td/osd-recovery-space/datafile 2023-11-02T09:42:32.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:32.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj528 td/osd-recovery-space/datafile 2023-11-02T09:42:32.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:32.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj529 td/osd-recovery-space/datafile 2023-11-02T09:42:32.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:32.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj530 td/osd-recovery-space/datafile 2023-11-02T09:42:32.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:32.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj531 td/osd-recovery-space/datafile 2023-11-02T09:42:32.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:32.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj532 td/osd-recovery-space/datafile 2023-11-02T09:42:32.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:32.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj533 td/osd-recovery-space/datafile 2023-11-02T09:42:32.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:32.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj534 td/osd-recovery-space/datafile 2023-11-02T09:42:32.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:32.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj535 td/osd-recovery-space/datafile 2023-11-02T09:42:32.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:32.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj536 td/osd-recovery-space/datafile 2023-11-02T09:42:32.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:32.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj537 td/osd-recovery-space/datafile 2023-11-02T09:42:32.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:32.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj538 td/osd-recovery-space/datafile 2023-11-02T09:42:33.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:33.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj539 td/osd-recovery-space/datafile 2023-11-02T09:42:33.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:33.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj540 td/osd-recovery-space/datafile 2023-11-02T09:42:33.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:33.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj541 td/osd-recovery-space/datafile 2023-11-02T09:42:33.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:33.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj542 td/osd-recovery-space/datafile 2023-11-02T09:42:33.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:33.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj543 td/osd-recovery-space/datafile 2023-11-02T09:42:33.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:33.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj544 td/osd-recovery-space/datafile 2023-11-02T09:42:33.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:33.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj545 td/osd-recovery-space/datafile 2023-11-02T09:42:33.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:33.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj546 td/osd-recovery-space/datafile 2023-11-02T09:42:33.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:33.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj547 td/osd-recovery-space/datafile 2023-11-02T09:42:33.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:33.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj548 td/osd-recovery-space/datafile 2023-11-02T09:42:33.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:33.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj549 td/osd-recovery-space/datafile 2023-11-02T09:42:33.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:33.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj550 td/osd-recovery-space/datafile 2023-11-02T09:42:33.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:33.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj551 td/osd-recovery-space/datafile 2023-11-02T09:42:33.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:33.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj552 td/osd-recovery-space/datafile 2023-11-02T09:42:34.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:34.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj553 td/osd-recovery-space/datafile 2023-11-02T09:42:34.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:34.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj554 td/osd-recovery-space/datafile 2023-11-02T09:42:34.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:34.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj555 td/osd-recovery-space/datafile 2023-11-02T09:42:34.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:34.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj556 td/osd-recovery-space/datafile 2023-11-02T09:42:34.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:34.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj557 td/osd-recovery-space/datafile 2023-11-02T09:42:34.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:34.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj558 td/osd-recovery-space/datafile 2023-11-02T09:42:34.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:34.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj559 td/osd-recovery-space/datafile 2023-11-02T09:42:34.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:34.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj560 td/osd-recovery-space/datafile 2023-11-02T09:42:34.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:34.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj561 td/osd-recovery-space/datafile 2023-11-02T09:42:34.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:34.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj562 td/osd-recovery-space/datafile 2023-11-02T09:42:34.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:34.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj563 td/osd-recovery-space/datafile 2023-11-02T09:42:34.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:34.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj564 td/osd-recovery-space/datafile 2023-11-02T09:42:34.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:34.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj565 td/osd-recovery-space/datafile 2023-11-02T09:42:34.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:34.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj566 td/osd-recovery-space/datafile 2023-11-02T09:42:34.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:34.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj567 td/osd-recovery-space/datafile 2023-11-02T09:42:35.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:35.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj568 td/osd-recovery-space/datafile 2023-11-02T09:42:35.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:35.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj569 td/osd-recovery-space/datafile 2023-11-02T09:42:35.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:35.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj570 td/osd-recovery-space/datafile 2023-11-02T09:42:35.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:35.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj571 td/osd-recovery-space/datafile 2023-11-02T09:42:35.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:35.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj572 td/osd-recovery-space/datafile 2023-11-02T09:42:35.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:35.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj573 td/osd-recovery-space/datafile 2023-11-02T09:42:35.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:35.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj574 td/osd-recovery-space/datafile 2023-11-02T09:42:35.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:35.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj575 td/osd-recovery-space/datafile 2023-11-02T09:42:35.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:35.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj576 td/osd-recovery-space/datafile 2023-11-02T09:42:35.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:35.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj577 td/osd-recovery-space/datafile 2023-11-02T09:42:35.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:35.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj578 td/osd-recovery-space/datafile 2023-11-02T09:42:35.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:35.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj579 td/osd-recovery-space/datafile 2023-11-02T09:42:35.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:35.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj580 td/osd-recovery-space/datafile 2023-11-02T09:42:35.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:35.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj581 td/osd-recovery-space/datafile 2023-11-02T09:42:35.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:35.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj582 td/osd-recovery-space/datafile 2023-11-02T09:42:35.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:35.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj583 td/osd-recovery-space/datafile 2023-11-02T09:42:36.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:36.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj584 td/osd-recovery-space/datafile 2023-11-02T09:42:36.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:36.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj585 td/osd-recovery-space/datafile 2023-11-02T09:42:36.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:36.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj586 td/osd-recovery-space/datafile 2023-11-02T09:42:36.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:36.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj587 td/osd-recovery-space/datafile 2023-11-02T09:42:36.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:36.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj588 td/osd-recovery-space/datafile 2023-11-02T09:42:36.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:36.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj589 td/osd-recovery-space/datafile 2023-11-02T09:42:36.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:36.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj590 td/osd-recovery-space/datafile 2023-11-02T09:42:36.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:36.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj591 td/osd-recovery-space/datafile 2023-11-02T09:42:36.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:36.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj592 td/osd-recovery-space/datafile 2023-11-02T09:42:36.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:36.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj593 td/osd-recovery-space/datafile 2023-11-02T09:42:36.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:36.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj594 td/osd-recovery-space/datafile 2023-11-02T09:42:36.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:36.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj595 td/osd-recovery-space/datafile 2023-11-02T09:42:36.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:36.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj596 td/osd-recovery-space/datafile 2023-11-02T09:42:36.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:36.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj597 td/osd-recovery-space/datafile 2023-11-02T09:42:37.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:37.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj598 td/osd-recovery-space/datafile 2023-11-02T09:42:37.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:37.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj599 td/osd-recovery-space/datafile 2023-11-02T09:42:37.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2023-11-02T09:42:37.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj600 td/osd-recovery-space/datafile 2023-11-02T09:42:37.213 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: expr 2 - 1 2023-11-02T09:42:37.214 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: seq 0 1 2023-11-02T09:42:37.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: for o in $(seq 0 $(expr $OSDS - 1)) 2023-11-02T09:42:37.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:122: TEST_recovery_test_simple: ceph tell osd.0 injectargs '--fake_statfs_for_testing 3686400' 2023-11-02T09:42:37.320 INFO:tasks.workunit.client.0.smithi116.stdout:{} 2023-11-02T09:42:37.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: for o in $(seq 0 $(expr $OSDS - 1)) 2023-11-02T09:42:37.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:122: TEST_recovery_test_simple: ceph tell osd.1 injectargs '--fake_statfs_for_testing 3686400' 2023-11-02T09:42:37.434 INFO:tasks.workunit.client.0.smithi116.stdout:{} 2023-11-02T09:42:37.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:124: TEST_recovery_test_simple: sleep 5 2023-11-02T09:42:42.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:126: TEST_recovery_test_simple: ceph pg dump pgs 2023-11-02T09:42:42.807 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T09:42:42.808 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 600 0 0 0 0 3072000 0 0 600 0 600 active+clean 2023-11-02T09:41:46.878799+0000 23'600 23:618 [1] 1 [1] 1 0'0 2023-11-02T09:41:44.126111+0000 0'0 2023-11-02T09:41:44.126111+0000 0 0 periodic scrub scheduled @ 2023-11-03T13:18:37.842675+0000 0 0 2023-11-02T09:42:42.808 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T09:42:42.808 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T09:42:42.809 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T09:42:42.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:128: TEST_recovery_test_simple: seq 1 1 2023-11-02T09:42:42.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:128: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2023-11-02T09:42:42.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:130: TEST_recovery_test_simple: ceph osd pool set test1 size 2 2023-11-02T09:42:43.465 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 size to 2 2023-11-02T09:42:43.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:134: TEST_recovery_test_simple: wait_for_recovery_toofull 30 2023-11-02T09:42:43.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:77: wait_for_recovery_toofull: local timeout=30 2023-11-02T09:42:43.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:78: wait_for_recovery_toofull: wait_for_state recovery_toofull 30 2023-11-02T09:42:43.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:52: wait_for_state: local state=recovery_toofull 2023-11-02T09:42:43.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:53: wait_for_state: local cur_in_state 2023-11-02T09:42:43.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: get_timeout_delays 30 5 2023-11-02T09:42:43.479 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-11-02T09:42:43.479 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-11-02T09:42:43.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-11-02T09:42:43.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-11-02T09:42:43.480 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-11-02T09:42:43.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: delays=('5' '10' '15') 2023-11-02T09:42:43.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: local -a delays 2023-11-02T09:42:43.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:55: wait_for_state: local -i loop=0 2023-11-02T09:42:43.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:57: wait_for_state: flush_pg_stats 2023-11-02T09:42:43.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-11-02T09:42:43.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-11-02T09:42:43.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-11-02T09:42:43.950 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2023-11-02T09:42:43.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-11-02T09:42:43.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:42:43.950 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-11-02T09:42:44.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836509 2023-11-02T09:42:44.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836509 2023-11-02T09:42:44.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836509' 2023-11-02T09:42:44.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-11-02T09:42:44.063 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-11-02T09:42:44.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672980 2023-11-02T09:42:44.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672980 2023-11-02T09:42:44.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836509 1-42949672980' 2023-11-02T09:42:44.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:42:44.246 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836509 2023-11-02T09:42:44.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:42:44.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-11-02T09:42:44.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836509 2023-11-02T09:42:44.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:42:44.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836509 2023-11-02T09:42:44.250 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836509 2023-11-02T09:42:44.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2023-11-02T09:42:44.251 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-11-02T09:42:44.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836509 -lt 21474836509 2023-11-02T09:42:44.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-11-02T09:42:44.766 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672980 2023-11-02T09:42:44.766 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-11-02T09:42:44.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-11-02T09:42:44.767 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672980 2023-11-02T09:42:44.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-11-02T09:42:44.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672980 2023-11-02T09:42:44.769 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672980 2023-11-02T09:42:44.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2023-11-02T09:42:44.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-11-02T09:42:45.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672980 -lt 42949672980 2023-11-02T09:42:45.329 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:58: wait_for_state: get_num_pgs 2023-11-02T09:42:45.329 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-11-02T09:42:45.329 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-11-02T09:42:45.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:58: wait_for_state: test 1 == 0 2023-11-02T09:42:45.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:62: wait_for_state: true 2023-11-02T09:42:45.908 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: get_num_in_state recovery_toofull 2023-11-02T09:42:45.908 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:43: get_num_in_state: local state=recovery_toofull 2023-11-02T09:42:45.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:44: get_num_in_state: local expression 2023-11-02T09:42:45.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:45: get_num_in_state: expression+='select(contains("recovery_toofull"))' 2023-11-02T09:42:45.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:46: get_num_in_state: ceph --format json pg dump pgs 2023-11-02T09:42:45.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:47: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("recovery_toofull"))] | length' 2023-11-02T09:42:46.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: cur_in_state=0 2023-11-02T09:42:46.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:64: wait_for_state: test 0 -gt 0 2023-11-02T09:42:46.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:65: wait_for_state: (( 0 >= 3 )) 2023-11-02T09:42:46.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:69: wait_for_state: sleep 5 2023-11-02T09:42:51.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:70: wait_for_state: loop+=1 2023-11-02T09:42:51.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:62: wait_for_state: true 2023-11-02T09:42:51.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: get_num_in_state recovery_toofull 2023-11-02T09:42:51.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:43: get_num_in_state: local state=recovery_toofull 2023-11-02T09:42:51.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:44: get_num_in_state: local expression 2023-11-02T09:42:51.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:45: get_num_in_state: expression+='select(contains("recovery_toofull"))' 2023-11-02T09:42:51.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:46: get_num_in_state: ceph --format json pg dump pgs 2023-11-02T09:42:51.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:47: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("recovery_toofull"))] | length' 2023-11-02T09:42:51.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: cur_in_state=0 2023-11-02T09:42:51.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:64: wait_for_state: test 0 -gt 0 2023-11-02T09:42:51.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:65: wait_for_state: (( 1 >= 3 )) 2023-11-02T09:42:51.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:69: wait_for_state: sleep 10 2023-11-02T09:43:01.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:70: wait_for_state: loop+=1 2023-11-02T09:43:01.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:62: wait_for_state: true 2023-11-02T09:43:01.670 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: get_num_in_state recovery_toofull 2023-11-02T09:43:01.670 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:43: get_num_in_state: local state=recovery_toofull 2023-11-02T09:43:01.670 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:44: get_num_in_state: local expression 2023-11-02T09:43:01.671 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:45: get_num_in_state: expression+='select(contains("recovery_toofull"))' 2023-11-02T09:43:01.671 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:46: get_num_in_state: ceph --format json pg dump pgs 2023-11-02T09:43:01.671 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:47: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("recovery_toofull"))] | length' 2023-11-02T09:43:02.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: cur_in_state=0 2023-11-02T09:43:02.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:64: wait_for_state: test 0 -gt 0 2023-11-02T09:43:02.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:65: wait_for_state: (( 2 >= 3 )) 2023-11-02T09:43:02.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:69: wait_for_state: sleep 15 2023-11-02T09:43:17.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:70: wait_for_state: loop+=1 2023-11-02T09:43:17.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:62: wait_for_state: true 2023-11-02T09:43:17.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: get_num_in_state recovery_toofull 2023-11-02T09:43:17.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:43: get_num_in_state: local state=recovery_toofull 2023-11-02T09:43:17.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:44: get_num_in_state: local expression 2023-11-02T09:43:17.050 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:45: get_num_in_state: expression+='select(contains("recovery_toofull"))' 2023-11-02T09:43:17.050 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:46: get_num_in_state: ceph --format json pg dump pgs 2023-11-02T09:43:17.050 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:47: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("recovery_toofull"))] | length' 2023-11-02T09:43:17.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: cur_in_state=0 2023-11-02T09:43:17.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:64: wait_for_state: test 0 -gt 0 2023-11-02T09:43:17.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:65: wait_for_state: (( 3 >= 3 )) 2023-11-02T09:43:17.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:66: wait_for_state: ceph pg dump pgs 2023-11-02T09:43:17.771 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T09:43:17.772 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 600 0 74 0 0 3072000 0 0 600 0 600 active+recovering+undersized+degraded+remapped 2023-11-02T09:42:44.636417+0000 23'600 30:1684 [1,0] 1 [1] 1 0'0 2023-11-02T09:41:44.126111+0000 0'0 2023-11-02T09:41:44.126111+0000 0 0 periodic scrub scheduled @ 2023-11-03T20:20:41.411124+0000 0 0 2023-11-02T09:43:17.772 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T09:43:17.772 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T09:43:17.772 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T09:43:17.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:67: wait_for_state: return 1 2023-11-02T09:43:17.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:136: TEST_recovery_test_simple: ERRORS=0 2023-11-02T09:43:17.783 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: ceph pg dump pgs 2023-11-02T09:43:17.783 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: grep +recovery_toofull 2023-11-02T09:43:17.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: wc -l 2023-11-02T09:43:18.131 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T09:43:18.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: '[' 0 '!=' 1 ']' 2023-11-02T09:43:18.143 INFO:tasks.workunit.client.0.smithi116.stdout:One pool should have been in recovery_toofull 2023-11-02T09:43:18.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:139: TEST_recovery_test_simple: echo 'One pool should have been in recovery_toofull' 2023-11-02T09:43:18.144 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:140: TEST_recovery_test_simple: expr 0 + 1 2023-11-02T09:43:18.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:140: TEST_recovery_test_simple: ERRORS=1 2023-11-02T09:43:18.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:143: TEST_recovery_test_simple: ceph pg dump pgs 2023-11-02T09:43:18.498 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-11-02T09:43:18.498 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 600 0 0 0 0 3072000 0 0 600 0 600 active+clean 2023-11-02T09:43:15.111138+0000 23'600 32:1843 [1,0] 1 [1,0] 1 0'0 2023-11-02T09:41:44.126111+0000 0'0 2023-11-02T09:41:44.126111+0000 0 0 periodic scrub scheduled @ 2023-11-03T19:49:15.483614+0000 0 0 2023-11-02T09:43:18.498 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T09:43:18.499 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-11-02T09:43:18.499 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2023-11-02T09:43:18.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:144: TEST_recovery_test_simple: ceph status 2023-11-02T09:43:19.000 INFO:tasks.workunit.client.0.smithi116.stdout: cluster: 2023-11-02T09:43:19.001 INFO:tasks.workunit.client.0.smithi116.stdout: id: e9475e6c-4956-4813-8dbd-b1d4bc7eb092 2023-11-02T09:43:19.001 INFO:tasks.workunit.client.0.smithi116.stdout: health: HEALTH_ERR 2023-11-02T09:43:19.001 INFO:tasks.workunit.client.0.smithi116.stdout: 2 full osd(s) 2023-11-02T09:43:19.001 INFO:tasks.workunit.client.0.smithi116.stdout: 1 pool(s) do not have an application enabled 2023-11-02T09:43:19.002 INFO:tasks.workunit.client.0.smithi116.stdout: 1 pool(s) full 2023-11-02T09:43:19.002 INFO:tasks.workunit.client.0.smithi116.stdout: OSD count 2 < osd_pool_default_size 3 2023-11-02T09:43:19.002 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T09:43:19.002 INFO:tasks.workunit.client.0.smithi116.stdout: services: 2023-11-02T09:43:19.002 INFO:tasks.workunit.client.0.smithi116.stdout: mon: 1 daemons, quorum a (age 2m) 2023-11-02T09:43:19.003 INFO:tasks.workunit.client.0.smithi116.stdout: mgr: x(active, since 2m) 2023-11-02T09:43:19.003 INFO:tasks.workunit.client.0.smithi116.stdout: osd: 2 osds: 2 up (since 97s), 2 in (since 2m) 2023-11-02T09:43:19.003 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T09:43:19.003 INFO:tasks.workunit.client.0.smithi116.stdout: data: 2023-11-02T09:43:19.004 INFO:tasks.workunit.client.0.smithi116.stdout: pools: 1 pools, 1 pgs 2023-11-02T09:43:19.004 INFO:tasks.workunit.client.0.smithi116.stdout: objects: 600 objects, 2.9 MiB 2023-11-02T09:43:19.004 INFO:tasks.workunit.client.0.smithi116.stdout: usage: 5.9 MiB used, 1.2 MiB / 7.0 MiB avail 2023-11-02T09:43:19.004 INFO:tasks.workunit.client.0.smithi116.stdout: pgs: 1 active+clean 2023-11-02T09:43:19.004 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T09:43:19.005 INFO:tasks.workunit.client.0.smithi116.stdout: io: 2023-11-02T09:43:19.005 INFO:tasks.workunit.client.0.smithi116.stdout: recovery: 126 KiB/s, 25 objects/s 2023-11-02T09:43:19.005 INFO:tasks.workunit.client.0.smithi116.stdout: 2023-11-02T09:43:19.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:145: TEST_recovery_test_simple: ceph status --format=json-pretty 2023-11-02T09:43:19.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:147: TEST_recovery_test_simple: jq .health.checks.PG_RECOVERY_FULL.severity td/osd-recovery-space/stat.json 2023-11-02T09:43:19.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:147: TEST_recovery_test_simple: eval SEV=null 2023-11-02T09:43:19.558 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:147: TEST_recovery_test_simple: SEV=null 2023-11-02T09:43:19.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:148: TEST_recovery_test_simple: '[' null '!=' HEALTH_ERR ']' 2023-11-02T09:43:19.559 INFO:tasks.workunit.client.0.smithi116.stdout:PG_RECOVERY_FULL severity null not HEALTH_ERR 2023-11-02T09:43:19.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:149: TEST_recovery_test_simple: echo 'PG_RECOVERY_FULL severity null not HEALTH_ERR' 2023-11-02T09:43:19.560 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:150: TEST_recovery_test_simple: expr 1 + 1 2023-11-02T09:43:19.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:150: TEST_recovery_test_simple: ERRORS=2 2023-11-02T09:43:19.560 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: jq .health.checks.PG_RECOVERY_FULL.summary.message td/osd-recovery-space/stat.json 2023-11-02T09:43:19.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: eval MSG=null 2023-11-02T09:43:19.561 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: MSG=null 2023-11-02T09:43:19.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:153: TEST_recovery_test_simple: '[' null '!=' 'Full OSDs blocking recovery: 1 pg recovery_toofull' ']' 2023-11-02T09:43:19.562 INFO:tasks.workunit.client.0.smithi116.stdout:PG_RECOVERY_FULL message 'null' mismatched 2023-11-02T09:43:19.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:154: TEST_recovery_test_simple: echo 'PG_RECOVERY_FULL message '\''null'\'' mismatched' 2023-11-02T09:43:19.564 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:155: TEST_recovery_test_simple: expr 2 + 1 2023-11-02T09:43:19.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:155: TEST_recovery_test_simple: ERRORS=3 2023-11-02T09:43:19.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:157: TEST_recovery_test_simple: rm -f td/osd-recovery-space/stat.json 2023-11-02T09:43:19.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:159: TEST_recovery_test_simple: '[' 3 '!=' 0 ']' 2023-11-02T09:43:19.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:161: TEST_recovery_test_simple: return 1 2023-11-02T09:43:19.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:36: run: return 1 2023-11-02T09:43:19.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: main: code=1 2023-11-02T09:43:19.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-recovery-space 1 2023-11-02T09:43:19.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2023-11-02T09:43:19.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2023-11-02T09:43:19.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2023-11-02T09:43:19.566 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-11-02T09:43:19.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-11-02T09:43:19.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-11-02T09:43:19.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-11-02T09:43:19.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-11-02T09:43:19.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-11-02T09:43:19.670 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-11-02T09:43:19.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-11-02T09:43:19.671 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-11-02T09:43:19.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-11-02T09:43:19.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-11-02T09:43:19.672 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-11-02T09:43:19.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:43:19.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-11-02T09:43:19.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-11-02T09:43:19.675 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-02T09:43:19.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:43:19.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2023-11-02T09:43:19.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2023-11-02T09:43:19.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2023-11-02T09:43:19.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/osd-recovery-space/mgr.x.log td/osd-recovery-space/mon.a.log td/osd-recovery-space/osd.0.log td/osd-recovery-space/osd.1.log /home/ubuntu/cephtest/archive/log 2023-11-02T09:43:19.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2023-11-02T09:43:19.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T09:43:19.687 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-11-02T09:43:19.687 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-11-02T09:43:19.687 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.316946 2023-11-02T09:43:19.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.316946 2023-11-02T09:43:19.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-11-02T09:43:19.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-11-02T09:43:19.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 1 2023-11-02T09:43:19.689 INFO:tasks.workunit:Stopping ['osd'] on client.0... 2023-11-02T09:43:19.689 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-11-02T09:43:19.966 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd/osd-recovery-space.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh' 2023-11-02T09:43:20.136 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3cd7d4f78c5144b1a09829c4d43539ce Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5a9486b4a6f144f83c51b7ec33111a2c86dc081/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd/osd-recovery-space.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh' 2023-11-02T09:43:20.141 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-02T09:43:20.158 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-02T09:43:20.158 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-02T09:43:20.240 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-02T09:43:20.251 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-02T09:43:20.252 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-02T09:43:20.266 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2023-11-02T09:43:20.269 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-02T09:43:20.270 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2023-11-02T09:43:20.270 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 14 +816us[ +845us] +/- 96ms 2023-11-02T09:43:20.270 INFO:teuthology.orchestra.run.smithi116.stdout:^* hv02.front.sepia.ceph.com 3 7 377 6 -864us[ -837us] +/- 52ms 2023-11-02T09:43:20.270 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 +453us[ +480us] +/- 59ms 2023-11-02T09:43:20.271 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-11-02T09:43:20.272 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-02T09:43:20.281 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-02T09:43:20.282 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-02T09:43:20.295 DEBUG:teuthology.orchestra.run.smithi116:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-02T09:43:20.400 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-02T09:43:20.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T09:43:20.432 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-02T09:43:20.444 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-02T09:43:20.453 INFO:teuthology.task.internal:Duration was 4941.588026 seconds 2023-11-02T09:43:20.454 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-02T09:43:20.466 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-02T09:43:20.466 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-02T09:43:20.515 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-02T09:43:20.898 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-02T09:43:20.899 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2023-11-02T09:43:20.899 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-02T09:43:20.959 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-02T09:43:20.960 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-02T09:43:21.028 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-02T09:43:21.029 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-02T09:43:21.161 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-02T09:43:21.184 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-02T09:43:21.185 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-02T09:43:21.214 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-02T09:43:21.228 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:43:21.279 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2023-11-02T09:43:21.307 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-02T09:43:21.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T09:43:21.363 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-02T09:43:21.375 INFO:teuthology.task.internal:Transferring archived files... 2023-11-02T09:43:21.377 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443892/remote/smithi116 2023-11-02T09:43:21.378 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-02T09:43:25.011 INFO:teuthology.task.internal:Removing archive directory... 2023-11-02T09:43:25.012 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-02T09:43:25.083 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-02T09:43:25.092 INFO:teuthology.task.internal:Not uploading archives. 2023-11-02T09:43:25.093 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-02T09:43:25.121 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-02T09:43:25.122 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-02T09:43:25.139 INFO:teuthology.orchestra.run.smithi116.stdout: 265854 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 2 09:43 /home/ubuntu/cephtest 2023-11-02T09:43:25.139 INFO:teuthology.orchestra.run.smithi116.stdout: 402521 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 2 08:28 /home/ubuntu/cephtest/mnt.0 2023-11-02T09:43:25.139 INFO:teuthology.orchestra.run.smithi116.stdout: 402522 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 2 09:40 /home/ubuntu/cephtest/mnt.0/client.0 2023-11-02T09:43:25.140 INFO:teuthology.orchestra.run.smithi116.stdout: 537967 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 2 09:40 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-02T09:43:25.140 INFO:teuthology.orchestra.run.smithi116.stdout: 537968 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 2 09:43 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2023-11-02T09:43:25.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-02T09:43:25.142 INFO:teuthology.orchestra.run.smithi116.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-11-02T09:43:25.143 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-11-02T09:43:25.147 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-02T09:43:25.165 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-02T09:43:25.181 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi116 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-11-02T09:43:25.186 INFO:teuthology.nuke:Checking targets against current locks 2023-11-02T09:43:25.226 DEBUG:teuthology.nuke:shortname: smithi116 2023-11-02T09:43:25.227 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-02T09:43:25.255 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-01_21:37:41-rados-wip-yuri6-testing-2023-11-01-0745-reef-distro-default-smithi/7443892', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-02 08:12:36.459141', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAZePFn69drfMB7+IWqDvyKDLk1QmpU3g2Wb7klHmM2zvqrmMq/zUBEdf8SMy931EUzCcIj8XyyQm5rFSAGqn/g='} 2023-11-02T09:43:25.320 INFO:teuthology.orchestra.console.smithi116:Power off 2023-11-02T09:43:25.322 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-02T09:43:25.357 DEBUG:teuthology.orchestra.console.smithi116:power off output: Chassis Power Control: Down/Off 2023-11-02T09:43:25.357 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-02T09:43:25.397 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2023-11-02T09:43:29.399 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-02T09:43:29.516 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2023-11-02T09:43:33.518 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-02T09:43:33.630 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is off 2023-11-02T09:43:33.731 INFO:teuthology.orchestra.console.smithi116:Power off completed 2023-11-02T09:43:33.842 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} duration: 4941.588025808334 failure_reason: 'Command failed (workunit test osd/osd-recovery-space.sh) on smithi116 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5a9486b4a6f144f83c51b7ec33111a2c86dc081 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3cd7d4f78c5144b1a09829c4d43539ce status: fail success: false 2023-11-02T09:43:33.843 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-02T09:43:33.928 INFO:teuthology.run:FAIL